Methods for processing event timing data

ABSTRACT

A method for processing a plurality of input images associated with a respective plurality of input times, the input images and input times being provided by an event timing system, comprises: selecting an output frame rate; generating a plurality of output images, corresponding to the output frame rate, from the plurality of input images; and assigning to each output image a final output time provided by the event timing system, the final output time being the input time associated with an input image contributing to the output image.

BACKGROUND

Photo finish cameras capture images of a finish line to accurately timestamp when a race participant crosses the finish line and to separatetwo or more race participants. Line scan cameras have proven useful asphoto finish cameras, because the finish line is projected onto thelength of a linear detector of the line scan camera. The line scancamera captures a series of consecutive images as race participantscross the finish line and then stitches them together to form atwo-dimensional representation of the participants. Since line scancameras contain only a single line of pixels, signal processing is fastand the cameras can operate at high frame rates. Accordingly, line scancameras provide high time resolution and therefore accurate timing ofevents.

SUMMARY

In an embodiment, a system processes event timing images and includes:area scan image sensor for generating sequential digital two-dimensionalimages of a scene; and time delay integration module for processing thesequential digital two-dimensional images to generate a time delayintegration image of a moving object in the scene.

In an embodiment, a method for processing event timing images comprises:capturing sequential digital two-dimensional images of a scene using anarea scan image sensor; and processing the sequential digitaltwo-dimensional images to generate a time delay integration image of anobject moving in the scene.

In an embodiment, a method for processing a plurality of input imagesassociated with a respective plurality of input times, the input imagesand input times being provided by an event timing system, comprises:selecting an output frame rate; generating a plurality of output images,corresponding to the output frame rate, from the plurality of inputimages; and assigning to each output image a final output time providedby the event timing system, the final output time being the input timeassociated with an input image contributing to the output image.

In an embodiment, a method for processing images and associated eventtimes provided by an event recording and timing system comprises:receiving (a) images and associated times and (b) a correspondencebetween times and events; selecting events of interest; andautomatically discarding images not associated with an event ofinterest, using a processor and machine readable instructions.

In an embodiment, a system is provided for recording and timing ofevents, and includes: a camera system for capturing images of the eventsand comprising a clock; an event recorder for detecting the events andbeing communicatively coupled with the clock; and a data processingsystem capable of assigning times provided by the clock to the imagescaptured by the camera system and events detected by the event recorder.

In an embodiment, an area scan image sensor includes: a plurality ofcolor pixels, each color pixel comprising three different photositetypes sensitive to three different colors, the photosites being arrangedin a 3×3 array such that each row and each column of 3×3 array comprisesthe three photosite types and every row and column has photositeconfiguration different from any other row and column, respectively.

In an embodiment, a system for processing event timing images includes:a camera comprising (a) an area scan image sensor for capturing imagesof a scene including a line and (b) a level; an adjustable mount coupledwith the camera; and an alignment control system for automaticallyadjusting the mount to align the camera with respect to the line.

In an embodiment, a system for processing event timing images includes:a camera comprising an image sensor for capturing images and a videogenerator for generating scoreboard type video; and a data processingmodule, communicatively coupled with the camera, for generating resultsdata from images received from the camera and communicating the resultsdata to the video generator.

In an embodiment, a software product includes instructions, stored onnon-transitory computer-readable media, wherein the instructions, whenexecuted by a computer, perform steps for processing sequential digitaltwo-dimensional images of a scene comprising a moving object to form atime delay integration image, and wherein the instructions includeinstructions for segmenting at least of portion of each of thesequential digital two-dimensional images into input lines; andinstructions for populating each line of the time delay integrationimage with an integral over a plurality of input lines, each of theplurality of input lines being selected from a different one of thesequential digital two-dimensional images to substantially match themovement of the moving object in a direction perpendicular to the inputlines.

In an embodiment, a software product includes instructions, stored onnon-transitory computer-readable media, wherein the instructions, whenexecuted by a computer, perform steps for processing a plurality ofinput images associated with a respective plurality of input times, theinput images and input times being provided by an event timing system,and wherein the instructions include: instructions for selecting anoutput frame rate; instructions for generating a plurality of outputimages, corresponding to the output frame rate, from the plurality ofinput images; and instructions for assigning to each output image afinal output time provided by the event timing system, the final outputtime being the input time associated with an input image contributing tothe output image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for processing event timing images,according to an embodiment.

FIG. 2 shows a schematic transform of consecutive images into a timedelay integration image performed by system of FIG. 1, according to anembodiment.

FIG. 3 illustrates a method for processing event timing images,according to an embodiment.

FIG. 4 illustrates a Bayer type color area scan image sensor forcapturing event timing images, according to an embodiment.

FIG. 5 illustrates a method for processing event timing images capturedby a Bayer type color area scan image sensor, according to anembodiment.

FIG. 6 illustrates an embodiment of the system of FIG. 1 for adjustingbrightness of TDI images, according to an embodiment.

FIG. 7 illustrates a method for adjusting brightness of time delayintegration images by varying the number of lines included in the timedelay integration, according to an embodiment.

FIG. 8 illustrates a method for improving the dynamic range of TDIimages by selecting the number of lines included in the time delayintegration on an individual pixel basis, according to an embodiment.

FIG. 9 illustrates a method for improving the dynamic range of TDIimages using fractional TDI, according to an embodiment.

FIG. 10A and FIG. 10B illustrate an area scan image sensor that includesa position dependent filter for providing image capture at differentbrightness level, according to an embodiment.

FIG. 11 illustrates a method for processing event timing images toadjust the brightness of a TDI image using the area scan image sensor ofFIG. 10A and FIG. 10B, according to an embodiment.

FIG. 12 illustrates a method for processing captured images to generatea TDI image with twice the resolution of the captured images, accordingto an embodiment.

FIG. 13 illustrates a method for processing images captured by a colorarea scan image sensor at double frame rate to generate a TDI image withtwice the resolution of the captured images, according to an embodiment.

FIG. 14 illustrates a Bayer type color area scan image sensor, whereindividual photosites are used to double the spatial resolution of acamera, according to an embodiment.

FIG. 15 illustrates a trilinear color image sensor, where individuallines of photo site are used to triple spatial resolution of a camera,according to an embodiment.

FIG. 16 illustrates a method for processing event timing images,according to an embodiment.

FIG. 17 illustrates a diagonal color filter array area scan imagesensor, wherein individual color pixels include a 3×3 photosite array,according to an embodiment.

FIG. 18 illustrates a method for processing event timing images capturedby a color area scan image sensor having color pixels withtwo-dimensional photosite variation, according to an embodiment.

FIG. 19 illustrates two exemplary color area scan image sensors havingmultiple regions with different color filter array properties, accordingto embodiments.

FIG. 20 illustrates a system for recording and optionally event timingimages, according to an embodiment.

FIG. 21 illustrates a system for processing event timing images using acamera and radio-frequency identification, according to an embodiment.

FIG. 22 illustrates a method for capturing event timing images,according to an embodiment.

FIG. 23 illustrates a method for cropping an image series to removeimages not associated with an event of interest, according to anembodiment.

FIG. 24 illustrates a method for processing event timing images,including generating time delay integration images of a moving object,according to an embodiment.

FIG. 25 illustrates integration and readout processes for the method ofFIG. 24, according to an embodiment.

FIG. 26 illustrates integration and readout processes for the method ofFIG. 24, according to an embodiment.

FIG. 27 illustrates one time delay integration camera system, accordingto an embodiment.

FIG. 28 illustrates a method for aligning the time delay integrationcamera system of FIG. 27, according to an embodiment.

FIG. 29 illustrates exemplary images captured by the time delayintegration camera system of FIG. 27 while performing the method of FIG.28, in an embodiment.

FIG. 30 illustrates a method for aligning the time delay integrationcamera of the system of FIG. 27, according to an embodiment.

FIG. 31 illustrates exemplary images captured by the time delayintegration camera system of FIG. 27 while performing the method of FIG.30, in an embodiment.

FIG. 32 illustrates a system for generating and displaying scoreboardvideo using a system for processing event timing images, according to anembodiment.

FIG. 33 illustrates one method for generating and displaying scoreboardvideo using a system for processing event timing images, according to anembodiment.

FIG. 34 illustrates yet another system for processing event timingimages, according to an embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Disclosed herein are systems and methods for processing event timingimages. In certain embodiments, such systems utilize area scan imagesensors that capture a plurality of two-dimensional images of a scene,such as the finish line area of a race. The plurality of two-dimensionalimages are processed by a time delay integration module, separate fromthe area scan image sensor, to generate a time delay integration (TDI)image. Since time delay integration is performed separately from theimage sensor, after image readout, the time delay integration processmay be flexibly optimized during or after image capture.

FIG. 1 illustrates one exemplary system 100 for processing event timingimages. System 100 is for example useful in an event to capture imagesof a runner, car, or other type of race participant (shown as object135) passing a finish line. System 100 includes an area scan imagesensor 110 for capturing digital two-dimensional images 115 of a scene130 imaged onto area scan image sensor 110 by imaging optics 120. System100 further includes a time delay integration (TDI) module 140, aninterface 150, and an optional clock 160 providing a time signal forarea scan image sensor 110 and TDI module 140. TDI module 140 includesimage processing circuitry 141. TDI module 140 optionally associates acapturing time 165, received from clock 160, with each digitaltwo-dimensional image 115. TDI module 140 receives the series ofconsecutive digital two-dimensional images 115(i) from area scan imagesensor 110 and processes image series 115(i) to provide TDI. Basedthereupon, TDI module 140 outputs a single, integrated TDI image 145,which provides a snapshot in time of object 135 as it moves throughscene 130.

In the prior art, TDI is performed on an image sensor as part of areadout process, and charges or voltages are added directly onboard theimage sensor before TDI images are read. In the embodiment of system100, on the other hand, image processing by TDI module 140 is performedexternally to image sensor 110 and after readout of images 115, as shownin FIG. 1, and involves post-processing of digital images 115 ratherthan changes to pixel voltages or charges at image sensor 110.

Imaging optics 120, area scan sensor 110, TDI module 140, interface 150,and optional clock 160 may be integrated into a camera 170. In analternative embodiment, TDI module 140 and interface 150 are implementedat an external device or computer (not shown in FIG. 1). Area scan imagesensor 110 is for example a CMOS area scan image sensor configured witha global or rolling shutter, where the latter may be implemented withglobal or rolling reset. Image processing circuitry 141 is for example afield programmable gate array (FPGA) configured to process image series115 and produce TDI image 145. Interface 150 communicates images to auser or an external computer and optionally facilitates control of TDImodule 140. In certain embodiments, interface 150 is a wirelessinterface such as a WiFi or Bluetooth interface.

In an embodiment, image processing circuitry 141 includesmachine-readable instructions encoded in non-volatile memory and aprocessor for executing these machine-readable instructions to generateTDI image 145 from image series 115. In another embodiment, imageprocessing circuitry 141 includes volatile memory, for receivingmachine-readable instructions encoded in non-volatile memory locatedelsewhere, and a processor for executing these machine-readableinstructions to generate TDI image 145 from image series 115. Forexample, volatile memory of image processing circuitry 141 may receivemachine-readable instructions from an erasable programmable read only(EPROM) memory or Flash memory coupled with system 100. In yet anotherembodiment, TDI module 140 includes machine-readable start-upinstructions encoded in non-volatile memory, for example in the form ofa boot record, which are executed upon starting system 100. The start-upinstructions include instructions for retrieving through interface 150,and loading to image processing circuitry 141, machine-readable imageprocessing instructions for processing digital two-dimensional images115(i), and optionally capturing times 165, as discussed above. Thestart-up instructions may be stored in flash memory within TDI module140. The image processing instructions are, for example, stored in anon-volatile memory within an external control system.

FIG. 2 shows one exemplary schematic transform 200 of consecutivedigital two-dimensional images 115 into TDI image 145 performed by TDImodule 140 using image processing circuitry 141. FIG. 3 shows oneexemplary method 300 for TDI used by TDI module 140 in this process.FIGS. 2 and 3 are best considered together in the following description.In FIG. 2, each line 146(j) in TDI image 145 is an integral of a seriesof lines 118(i,j), where each line 118(i,j) is extracted from adifferent two-dimensional image 115(i). In an embodiment, the frame rateat which two-dimensional images 115(i) are captured, and the pitchbetween adjacent lines 118(i,j) measured in image space, are set tosubstantially match the speed of an object of interest moving throughscene 130 (such that object 135 progresses through successive lines118(i,j) at a rate of one line per frame). A “line” as used hereinrefers to image data formed by or into a single column or row of a twodimensional image. In a use scenario concerned with the timing of raceparticipants crossing a finish line, lines 118 are advantageouslyoriented to be substantially parallel to the image of the finish line.It is further beneficial to align system 100 such that the finish lineimage coincides with given line 118.

In a step 310 of method 300, TDI module 140 segments each digitaltwo-dimensional image 115 within the image series 115(i) into lines118(i,j), where i indicates the image and j indicates the line withinthat image (not all lines are labeled in FIG. 2 for clarity ofillustration). In one embodiment, TDI module 140 segments images 115into lines 118, where images 115 are received by TDI module 140 from anexternal source in the form of two-dimensional images. In anotherembodiment, TDI module 140 receives images 115 in the form of lines 118and step 310 is executed as an integral part of the receiving process.

TDI image 145 is also composed of multiple lines 146(i) (though onlyline 146(7) is labeled in FIG. 2 for clarity of illustration). Inparticular, in a step 320, TDI module 140 forms a series of lines118(i,j), where each line is extracted from a different image 115(i). Asobject 135 moves through scene 130, the captured image of object 135moves across lines 118(i,j). For example, as shown in FIG. 2, aparticular portion of object 135 (the runner's torso) is located in line118(1,6) in image 115(1), in line 118(2,7) in image 115(2), and in line118(3,8) in image 115(3). The series of lines 118(i,j) formed in step320 tracks the movement of object 135 as it progresses throughsequentially captured images 115(i).

In a step 330, TDI module 140 integrates the series of lines 118(i,j) toform a single, integrated line 146(j). For example, the n'th pixel ofthe integrated line is the sum of all n'th pixels in the series of lines118(i,j). In a step 340, line 146(j) of TDI image 145 is set to equalthe integrated line generated in step 330. Using the example of therunner's torso in FIG. 2, line 118(1,6) from image 115(1), line 118(2,7)from image 115(2), and line 118(3,8) from image 115(3) are integrated instep 330 to form a single, integrated line 146(7). Step 330 may utilizefractional TDI, wherein the single, integrated line 146(7) is theintegral of a non-integer number of lines 118(i,j). For example, line146(7) may be formed as the line 118(1,6)+line 118(2,7)+x line 118(3,8),where x is a number greater than zero and smaller than one. FractionalTDI is discussed further in connection with method 900 of FIG. 9.

Steps 320 through 340 are repeated until all lines 146(j) of TDI image145 have been generated. Note that different subsets of an image series115(i) may be used to generate different lines 146(j) of TDI image 145.

In a step 360, the TDI image is outputted. For example, TDI module 140(FIG. 1) outputs TDI image 145 to interface 150. Interface 150 may beconnected to an external system, such as an external processing system,whereby TDI module 145 outputs the TDI image to the external processingsystem.

In the example in FIG. 2, three consecutive images 115(1), 115(2), and115(3) are processed by TDI module 140 to generate TDI image 145. Asobject 135 moves through scene 130, the position of object 135 shifts byone line for each successive frame. A particular portion of object 135(the runner's torso) is located in line 118(1,6) in image 115(1), inline 118(2,7) in image 115(2), and in line 118(3,8) in image 115(3).Line 146(7) of TDI image 145 is the integral of lines 118(1,6) of image115(1), 118(2,7) of image 115(2), and 118(3,8) of image 115(3). Althoughsimilar to image 115(3), TDI image 145 shows object 135 with greaterbrightness and improved signal-to-noise ratio. Accordingly, images maybe captured by area scan image sensor 110 at a higher frame rate tomatch the speed of a faster moving object, or in inferior lightingenvironments, as compared to systems not utilizing TDI processing ofFIGS. 1-3. Without TDI processing, slower frame rates are required tocapture images of sufficient sensitivity to identify objects ofinterest, resulting in coarser time resolution.

As illustrated in FIG. 2, TDI image 145 is suitable for timing of object135 passing a line, such as a finish line, that is located to correspondto line 146(7) of TDI image 145. Accordingly, TDI image 145 may be givena timestamp that is the time associated with image 115(2), such as thecapture time of image 115(2). The example thus illustrated in FIG. 2 maybe modified for timing of object 135 passing a line that corresponds toanother line 146(i) of TDI image 145, without departing from the scopehereof. For example, line 146(8) of TDI image 145 may be populated withthe integral of lines 118(1,6) of image 115(1), 118(2,7) of image115(2), and 118(3,8) of image 115(3), and the resulting TDI image 145may be given a timestamp that is the time associated with image 115(3).Generally, the integrals contributing to TDI image 145 may be performedwith any line 146(i) of TDI image 145 corresponding to a finish line, orother timing line. Consistent therewith, TDI image 145 may be given atimestamp that is the time of any image 115 contributing to TDI image145.

Image processing circuitry 141 of TDI module 140 (FIG. 1) may beconfigured to process images 115(i) by assuming a direction of movementof objects passing through the scene, as discussed in connection withFIGS. 2 and 3. The processing performed by image processing circuitry141 of TDI module 140 may also be adapted to optimize for differentdirections of object movement. In one embodiment, area scan image sensor110 is implemented as a rectangular array of pixels, and lines 118(i,j)of images 115 are naturally oriented to coincide with either rows orcolumns of pixels of area scan image sensor 110. For each of these twoorientations of lines 118(i,j), images 115 are processed to optimize forobject movement in either of the two directions perpendicular to lines118(i,j). An image series 115(i) may be processed by image processingcircuitry 141 of TDI module 140 in several different ways to provideseveral different TDI images, each optimized for different directions ofobject movement.

In one embodiment, lines 118(i,j) of image 115(i) represent the fullnumber of either rows or columns of area scan image sensor 110,implemented as a rectangular array of pixels, corresponding to using thefull active area of area scan image sensor 110. In another embodiment,images 115 include only a portion of the active area, such that lines118(i,j) of image 115(i) represent only a subset of the rows and/orcolumns of area scan image sensor 110. In yet another embodiment, image115 includes the full active area but only a portion thereof is utilizedby TDI module 140, such that lines 118(i,j) of image 115(i) representonly a subset of the rows and/or columns of area scan image sensor 110.

In certain embodiments, steps 320 through 360 are repeated for twodifferent, non-contiguous portions of the active area of area scan imagesensor 110 to generate two respective TDI images representative ofdifferent subsets of a scene. For example, area scan image sensor 110and imaging optics 120 may be aligned such that the finish line of arace intersects the optical axis of imaging optics 120. One selectedportion of images 115 captured by area scan image sensor 110 of a scene130 shows race participants crossing the finish line of a race, whileanother selected portion of images 115 shows race participants crossinga secondary “pre-finish line”, located before the actual finish line. Inthe TDI image generated from the finish line portion of images 115, raceparticipants may occlude each other. Since the pre-finish line does notintersect the optical axis of imaging optics 120, the TDI imagegenerated from the pre-finish line portion of images 115 will show raceparticipants in a more frontal view. Race participants may therefore bemore easily separated in the view provided by the pre-finish line TDIimage.

The systems and methods of FIGS. 1-3 differ from prior artcharge-coupled devices (CCD) systems directly outputting a TDI image. Inthe prior art CCD, photo-induced charges accumulated at different times,and at different locations on the CCD, are integrated as part of asynchronized image sensor readout process to generate the TDI image. Incontrast, the systems and methods of FIGS. 1-3 are based on generationof two-dimensional images and the image pixel values of thesetwo-dimensional images are processed outside the image sensor togenerate a TDI image, enabling post-capture optimization of a variety ofaspects of a TDI image. Such aspects include, but are not limited to,TDI image brightness, dynamic range, sharpness, noise level, andresolution (see, for example, FIGS. 4-18). Furthermore, the presentlydisclosed systems and methods for generating TDI images generate and/orutilize two-dimensional images that may be used for other purposes thanTDI, such as camera alignment (FIGS. 27-31) or video generation (FIGS.32-34); and different portions of the two-dimensional images generatedmay be processed and/or utilized differently (see, for example, FIGS. 3,10, 11, and 19).

FIG. 4 illustrates one exemplary Bayer type color area scan image sensor400. Color area scan image sensor 400 is an embodiment of area scanimage sensor 110 of FIG. 1. Color area scan image sensor 400 includes aBayer type pixel array. In the present disclosure, a Bayer type pixelarray is a type of pixel array wherein each color pixel is composed ofone first-type photosite sensitive to a first color, one second-typephotosite sensitive to a second color, and two third-type photositessensitive to a third color. Each color pixel 420 of color area scanimage sensor 400 is composed of four photosites 421, 422, 423, and 424.In an embodiment, photosite 421 is sensitive to red (R) light,photosites 422 and 423 are sensitive to green (G) light, and photosite424 is sensitive to blue (B) light. Color area scan image sensor 400 isillustrated in FIG. 4 as having three lines 410(1), 410(2), and 410(3)of color pixels. In an embodiment, lines 410 are oriented substantiallyperpendicular to the direction of motion of an object, for exampleobject 135 (FIG. 1). Each line 410 includes multiple color pixels 420.Only one color pixel 420 is illustrated for each line 410 in FIG. 4.Line 410(1) includes a color pixel 420(1), line 410(2) includes a colorpixel 420(2), and line 410(3) includes a color pixel 420(3). Colorpixels 420(1), 420(2), and 420(3) are located at the same verticalposition within corresponding lines 410(2), 410(2), and 410(3). As theobject travels, substantially the same portion of the object may beimaged by each of color pixels 420(1), 420(2), and 420(3) as timeprogresses. For example, color area scan image sensor 400 may captureimages at a frame rate that matches the speed with which the objectmoves through the frame, as discussed in connection with FIGS. 2 and 3.Color area scan image sensor 400 may be composed of more than threelines 410 without departing from the scope hereof. Likewise, photosites421, 422, 423, and 424 may be arranged differently within color pixel420, without departing from the scope hereof. For example, the locationsof two or more of photosites 421, 422, 423, and 424 may be swapped ascompared to the illustration of FIG. 4. An exemplary direction ofmovement of an object is indicated by arrow 430.

FIG. 5 illustrates one exemplary method 500 for generating a TDI imagefrom images captured by a color area scan image sensor having a Bayertype pixel array. Method 500 may be extended to generating a TDI imagefrom images captured by a color area image sensor where each color pixelis composed of a two-by-two photosite array, without departing from thescope hereof. Method 500 is an embodiment of method 300 (FIG. 3)applicable to generation of TDI images by system 100 (FIG. 1) with colorarea scan image sensor 400 of FIG. 4 implemented as area scan imagesensor 110 (FIG. 1). Method 500 assumes that the color area scan imagesensor captures images 115 (FIGS. 1 and 2) at a frame rate such that anobject of interest progresses through lines 118 (FIG. 2) of the colorarea scan image sensor at a rate of one line per frame. For example,color area scan image sensor 400 (FIG. 4) captures images of scene 130(FIG. 1) at a rate such that object 135 (FIG. 1) progresses throughlines 410 (FIG. 4) at a rate of one line per frame in the directionindicated by arrow 430 (FIG. 4). Method 500 is performed, for example,by TDI module 140 (FIG. 1).

In a step 510, each two-dimensional image captured by the color areascan image sensor is received in the form of rows. The rows are orientedparallel with the lines of method 300 (FIG. 3), such that a line ofmethod 300 corresponds to two rows of method 500. The two rows are anR&G row composed of signals from R and G photosites and a G′&B rowcomposed of signals from G′ and B photosites. For example, TDI module140 (FIG. 1) receives two-dimensional images 115 (FIG. 1) captured bycolor area scan image sensor 400 (FIG. 4) as rows, such that each line410 (FIG. 4) is associated with two rows: (a) a row composed of all R1(421(1)) and G1 (422(1)) photosite signals from line 410 and (b) a rowcomposed of all G1′ (423(1)) and B1 (424(1)) photosite signals from line410. In another example, TDI module 140 (FIG. 1) receivestwo-dimensional images 115 (FIG. 1), captured by color area scan imagesensor 400 (FIG. 4), in an arbitrary format. TDI module 140 (FIG. 1)processes the two-dimensional images 115 (FIG. 1) to generate rows, suchthat each line 410 (FIG. 4) is associated with two rows: (a) a rowcomposed of all R1 and G1 photosite signals from line 410 and (b) a rowcomposed of all G1′ and B1 photosite signals from line 410.

Following step 510, method 500 proceeds to populate each line of the TDIimage by performing steps 521, 522, 531, 532, and 540 for each line inthe TDI image. Steps 521 and 531 are performed sequentially, as aresteps 522 and 532. Sequential steps 521 and 531 may be performed inparallel or series with sequential steps 522 and 532. In step 521, aseries of R&G rows is formed, wherein each R&G row is extracted from adifferent image. The series of R&G rows follows the progression of anobject through a scene, as discussed in connection with FIGS. 2 and 3.For example, TDI module 140 (FIG. 1) forms a series of R&G rowsassociated with the respective series of lines 410(1), 410(2), and410(3) of color area scan image sensor 400 (FIG. 4). The series of R&Grows are extracted from a respective series of sequentially capturedimages 115 (FIG. 1), where images 115 are captured at a frame rate suchthat an object 135 (FIG. 1) moves through the frame at a rate of oneline 410 per frame. In step 531, the series of R&G rows generated instep 521 is integrated to form a single, integrated R&G row. Forexample, TDI module 140 (FIG. 1) integrates the series of R&G rowsgenerated in step 521 to form a single, integrated R&G row. In step 522,a series of G′&B rows, each from a different image, is formed. Theseries of G′&B rows follows the progression of an object through ascene, as discussed in connection with FIGS. 2 and 3. For example, TDImodule 140 (FIG. 1) forms a series of G′&B rows associated with therespective series of lines 410(1), 410(2), and 410(3) of color area scanimage sensor 400 (FIG. 4). The series of G′&B rows are extracted from arespective series of sequentially captured images 115 (FIG. 1), whereimages 115 are captured at a frame rate such that an object 135 (FIG. 1)moves through the frame at a rate of one line 410 per frame. In step532, the series of G′&B rows generated is step 522 is integrated to forma single, integrated G′&B row. For example, TDI module 140 (FIG. 1)integrates the series of G′&B rows generated in step 522 to form asingle, integrated G′&B row.

In step 540, the single, integrated R&G row generated in step 531 iscombined with the single, integrated G′&B row generated in step 532 toform a single color pixel line. This color pixel line includes thecombined R, G, G′, and B color data and forms a line of a TDI image. Forexample, TDI module 140 (FIG. 1) combines the R&G row with the G′&B row.In one embodiment of step 530, the combination is performed such thateach color pixel of the TDI line is represented by a quadruplet composedof the four individual R, G, G′, B values. In another embodiment, thecombination is performed such that each color pixel of the TDI line isrepresented by a triplet composed of three individual values: R, theaverage of G and G′, and B. In yet another embodiment, the combinationis performed such that each color pixel of the TDI line is representedby two triplets: one triplet composed of R, G, and B values and onetriplet composed of R, G′, and B values. Following step 540, method 500proceeds to perform step 360 of method 300 (FIG. 3).

While area scan image sensor 400 of FIG. 4 and method 500 of FIG. 5 arediscussed in the context of a Bayer type color filter array, both areascan image sensor 400 and method 500 may be extended to non-Bayer typecolor filter arrays without departing from the scope hereof. In oneembodiment, photosites 421, 422, 423, and 444 (FIG. 4) are sensitive tofour different colors, such that G′ represents a color different from G.This corresponds to an embodiment of method 500 (FIG. 5) wherephotosites G′ correspond to a color different from photosites G. Areascan image sensor 400 (FIG. 4) and method 500 (FIG. 5) may be furtherextended to color filter arrays having other geometrical photositeconfigurations. For example, each color pixel may be composed of alarger number of photosites, such as 3×3 photosites. In another example,each color pixel has a number of photosites that are vertically stackedas in the Foveon X3 sensor from Foveon, Inc.

FIG. 6 illustrates one exemplary system 600 for processing event timingimages, which is an embodiment of system 100 of FIG. 1. System 600incorporates functionality for adjusting the brightness of TDI images byadjustment of one or more of several parameters, including the number oflines 118 (FIGS. 1 and 2) in a series of sequentially capturedtwo-dimensional images 115 (FIGS. 1 and 2) used to generate each line146 (FIG. 2) of the TDI image 145 (FIGS. 1 and 2).

System 600 includes area scan image sensor 610, imaging optics 620, andTDI module 640, which are embodiments of area scan image sensor 110,imaging optics 120, and TDI module 140, respectively, of system 100(FIG. 1). Area scan image sensor 610 includes sensor settings 630.Sensor settings 630 include a gain setting 632 that defines theelectronic gain of area scan image sensor 610, a frame rate setting 634that defines the rate at which area scan image sensor 610 capturesframes, for example images 115 (FIGS. 1 and 2), and an exposure timesetting 636 that defines the exposure time for images captured by areascan image sensor 610.

Imaging optics 620 includes an optional, adjustable aperture 622, suchas an iris, that affects the amount of light transported through imagingoptics 620. Thus, optional aperture 622 may be adjusted to achieve acertain brightness of the image formed by imaging optics 620 on areascan image sensor 610. Optionally, imaging optics 620 further includes aconfigurable filter 624. In an embodiment, configurable filter 624includes one or more of (a) an infrared filter portion for blocking atleast a portion of infrared light from reaching area scan image sensor610, (b) one or more neutral density filters for reducing the amount oflight transmitted by imaging optics 620, and (c) a blank filter fortransmitting light without filtering. In an embodiment, configurablefilter 624 is motorized and may be controlled by an electrical controlsignal.

TDI module 640 further includes image processing circuitry 641 as anembodiment of image processing circuitry 141 (FIG. 1). Image processingcircuitry 641 includes an optional line number setting 642, an optionalceiling value 644, and an optional digital gain setting 646. Line numbersetting 642 is the number of lines 118 in images 115 (FIGS. 1 and 2).Line number setting 642 may be set to the maximum number of lines thatarea scan image sensor 610 can provide, or a subset thereof. Ceilingvalue 644 is a ceiling for the value that a single TDI image pixel mayachieve during the generation thereof. Digital gain setting 646 definesa digital gain applied to images 115 (FIGS. 1 and 2) and/or a TDI imagegenerated therefrom. TDI module 640 may utilize line number setting 642and/or digital gain setting 646 in conjunction with method 700 of FIG.7, discussed below, to adjust the brightness of TDI images 145. TDImodule 640 may further utilize digital gain setting 646 in conjunctionwith method 900 of FIG. 9, discussed below, to perform fractional TDI.TDI module 640 may utilize ceiling value 644 in conjunction with method800 of FIG. 8, discussed below, to adjust the brightness of TDI images145 on an individual pixel basis, and thereby improve the dynamic rangeof TDI images 145.

In certain embodiments, system 600 includes clock 160 (FIG. 1) forcommunicating a time signal, such as capturing time 165, to area scanimage sensor 610, TDI module 640, interface 150, and a controller 650.Area scan image sensor 610, imaging optics 620, TDI module 640,controller 650, interface 150, and optional clock 160 may be integratedinto a camera 670. Camera 670 is an embodiment of camera 170 (FIG. 1).

Controller 650 communicates control signals 615, 625, and 645 to areascan image sensor 610, imaging optics 620, and TDI module 640,respectively. Control signal 615 adjusts gain setting 632, frame ratesetting 634, and, optionally, exposure time 636. In an embodiment, areascan image sensor 610 is configured to maximize exposure time setting636 for images 115 (FIGS. 1 and 2) given a frame rate setting 634. Inthis case, exposure time setting 636 is approximately the inverse valueof frame rate setting 634. Therefore, frame rate setting 634 definesexposure time setting 636 and may be adjusted to adjust the brightnessof images 115 (FIGS. 1 and 2). In an alternative embodiment exposuretime setting 636 may be reduced compared to the maximum exposure timeassociated with a given frame rate setting 634. In this case, controlsignal 615 may adjust the exposure time directly.

In an embodiment, control signal 625 adjusts one or both of aperture 622and configurable filter 624. For example, control signal 625 adjusts thediameter of aperture 622 to adjust the brightness of images formed onarea scan image sensor 610. In another example, control signal 625adjusts which portion of configurable filter 624 is in the light path.

In another embodiment, control signal 645 adjusts line number setting642, ceiling value 644, and/or digital gain 646. The number of lines 118used to generate a TDI line 146 is a parameter that may be adjusted toadjust the brightness of object 135 (FIGS. 1 and 2) in TDI image 145.For example, the brightness of object 135 in a TDI line 146 generatedfrom twenty lines 118 is twice that achieved in a TDI line 146 generatedfrom ten lines 118. Likewise, ceiling value 644, as discussed below inconnection with FIG. 8, is a parameter that may be adjusted to adjustthe appearance, by local brightness adjustment, of object 135 (FIGS. 1and 2) in TDI image 145. Digital gain setting 646 is a parameter thatmay be used to globally or locally adjust brightness of TDI image 145.

Accordingly, system 600 provides eight adjustable parameters forachieving a certain brightness of TDI image 145: gain setting 632, framerate setting 634, exposure time 636, size of aperture 622, setting ofconfigurable filter 624, line number setting 642, ceiling value 644, anddigital gain 646. Generally, gain setting 632, frame rate setting 634,and the size of aperture 622 all impact properties of images 115 otherthan brightness. In certain embodiments, exemplified by the illustrationin FIG. 2, frame rate setting 634 is fixed in order to match the rate ofmovement of object 135 to the spacing of lines 118 of images 115. Hence,the frame rate setting 634 is not available for brightness adjustment.Exposure time setting 636 is upwards-limited by frame rate setting 634and may not be available to increase the brightness. Typically, gainsetting 632 affects the noise level of images 115, such that anincreased value of gain setting 632 is associated with increased noisein images 115.

In use scenarios where the brightness of object 135 (FIG. 1) is toohigh, for example so high that portions of an image captured by areascan image sensor 610 is saturated, the setting of configurable filter624 may be adjusted to reduce the amount of transmitted light. However,such adjustment can typically be made only in discrete increments.Exposure time setting 636 and/or digital gain setting 646 may be used insuch a scenario to more finely adjust the brightness of object 135 (FIG.1), for example in conjunction with adjusting the setting ofconfigurable filter 624.

In use scenarios where the brightness of object 135 in the image formedon area scan image sensor 610 is low, the range of gain setting 632 maybe limited to a range that produces images 115 of a requiredsignal-to-noise ratio. The size of aperture 622 is, in most opticalimaging systems, related to the depth of focus. The size of aperture 622may be increased in order to increase the brightness of image 115 (FIGS.1 and 2); however, this decreases the depth of focus. Thus, in some usescenarios, the size of aperture 622 is upwards-limited by depth of focusrequirements. Digital gain setting 646 may be used to increase thebrightness of object 135; however, digital gain setting 646 will,generally, affect signal and noise equally such that the signal-to-noiseratio is unimproved. It is therefore advantageous to be able to adjustthe brightness of TDI image 145 through line number setting 642, eitheralone or in combination with adjustment of one or more of gain setting632, frame rate setting 634, exposure time setting 636, digital gainsetting 646, the size of aperture 622, and setting of configurablefilter 624. As an alternative to adjusting the brightness through linenumber setting 642, the brightness may be adjusted through ceiling value644. Ceiling value 644 facilitates a local brightness adjustment, whichmay be used to improve the dynamic range of the TDI image 145 inaddition to adjusting the overall brightness of TDI image 145.

Controller 650 may communicate with TDI module 640, area scan imagesensor 610, and, optionally, imaging optics 620 through interface 150,without departing from the scope hereof. Likewise, all or portions ofthe functionality of controller 650 may be placed externally to system600 and be communicatively coupled to system 600 through interface 150,without departing from the scope hereof.

FIG. 7 illustrates one exemplary method for adjusting the brightness ofTDI image 145 (FIGS. 1 and 2) using system 600 of FIG. 6. In a step 710,the line number setting 642 is set to a certain value. For example,controller 650 (FIG. 6) communicates control signal 645 (FIG. 6) to TDImodule 640 (FIG. 6). In certain embodiments, line number setting 642 isa non-integer value as discussed in connection with method 900 (FIG. 9).In an optional step 720, the values of one or more of gain setting 632(FIG. 6), frame rate setting 634 (FIG. 6), exposure time setting 636(FIG. 6), digital gain setting 646 (FIG. 6), the size of aperture 622(FIG. 6), and setting of configurable filter 624 (FIG. 6) are adjusted.For example, controller 650 (FIG. 6) communicates control signals 615(FIG. 6) and/or 625 (FIG. 6) to area scan image sensor 610 and imagingoptics 620, respectively. The order of step 710 and optional step 720may be reversed, or step 710 and optional step 720 may be executed inparallel. In a step 730, a TDI image 145 (FIGS. 1 and 2) is generated.For example, a series of images 115(i) (FIGS. 1 and 2) are captured byarea scan image sensor 610 (FIG. 6), and processed by TDI module 640(FIG. 6), according to method 500 (FIG. 5), using image processingcircuitry 641.

In an optional step 740, the brightness of TDI image 145 is evaluated.Based on the result of the evaluation, method 700 may return to step 710for further brightness adjustment. In one embodiment, step 740 isperformed automatically by controller 650 or by a computer externally tosystem 600. In another embodiment, step 740 is performed manually by anoperator.

In certain embodiments, the parameters available for brightnessadjustment in steps 720 and 730 are associated with one or more oftarget value, minimum value, and maximum value. Further, each of theparameters may be assigned a priority such that method 700 is performedaccording to a specified sequence of parameter adjustments. Method 700may be performed automatically and/or by an operator.

In an exemplary use scenario, images are captured during sunset suchthat the environment steadily loses light and method 700 isautomatically performed to increase TDI image brightness. Gain setting632 may initially be at a specified target value. In order to increasebrightness, method 700 may first increase the size of aperture 622 to aspecified maximum value, for example a value known to not adverselyaffect other image properties. If adjustment is insufficient, method 700may proceed to increasing the value of line number setting 642 to aspecified maximum value. If this also proves insufficient, method 700may, after approval by an operator, increase gain setting 632 beyond aspecified target value to its maximum value.

None of the steps of method 700 require an area scan image sensor orarea scan images. Hence, method 700 may be extended to systemsequivalent to system 600 of FIG. 6, with area scan image sensor 610replaced by another type of image sensor such as a line scan imagesensor, without departing from the scope hereof. Furthermore, theoperations of step 720 may be applied to a camera based on an area scanimage sensors or a non-area scan image sensor, such as a line scan imagesensor, where these cameras are used in applications that do not includeTDI. In this case, the captured images are directly evaluated to guidethe performance of step 720.

FIG. 8 illustrates one exemplary method 800 for providing an improveddynamic range of a TDI system. Method 800 is an extension of method 700(FIG. 7), wherein the number of lines used to generate a TDI image isautomatically determined on an individual pixel basis. A scene, forexample scene 130 (FIG. 1), imaged by a TDI system may include brightareas and dim areas. The object of interest, such as object 135 (FIG.1), may be significantly less bright than other objects that are not ofinterest. Likewise, some portions of the object of interest may be muchbrighter than other portions thereof. Method 800 allows for utilizingmore lines when populating pixels of the TDI image associated with darkareas and fewer lines when populating pixels of the TDI image associatedwith bright areas.

Generally, noise is more apparent in a dark area, while blur is moreapparent in a bright area. Blur may result from a mismatch between theimage capture frame rate and the local or global movement rate of theobject. Hence, for a given bright portion of an object of interest, thenumber of lines used to generate a pixel of the TDI image isadvantageously kept low in order to minimize the amount of potentialblur associated with mismatch between the image capture frame rate andthe rate of movement of the given object portion. For a dim portion ofan object of interest, the number of lines used in the TDI process isadvantageously increased in order to increase the signal-to-noise ratio.Method 800 increases the dynamic range of a TDI image beyond the dynamicrange of the image sensor used to capture the images, from which the TDIimage is generated. Accordingly, method 800 may be advantageouslyutilized by an event timing system to ensure high quality TDI images.

Method 800 is performed, for example, by TDI module 140 of system 100(FIG. 1), or by TDI module 640 of system 600 (FIG. 6) using ceilingvalue 642 (FIG. 6).

In an exemplary scenario, scene 130 is a finish area of a night timehorse race, object 135 is a racing horse, and the finish area isilluminated by stadium lighting. Some portions of the racing horseappear very bright due to stadium light reflections off of the racinghorse. Other portions of the racing horse, which are in a shadow, appearvery dark. Method 800 provides for increasing the dynamic range of theTDI image beyond the dynamic range of the image sensor used for imagecapture, such that bright portions of the racing horse appear withminimal blur while dark portions of the racing horse appear with maximumsignal-to-noise ratio.

In a step 810, a ceiling value for the brightness of a TDI pixel isreceived. For example, TDI module 640 (FIG. 6) receives a ceiling valuefrom interface 150 (FIGS. 1 and 6)) through controller 650 and storesthis ceiling value to ceiling value 644 (FIG. 6). Following step 810,method 800 performs steps 820, 830, and 840 for each pixel of the TDIimage.

In step 820, the initial value of the TDI image is set to the value of acorresponding pixel in one of the captured images. For example, TDImodule 640 (FIG. 6) sets the initial value of the TDI pixel to the valueof a corresponding pixel from one of images 115 (FIGS. 1 and 6) receivedfrom area scan image sensor 110 (FIGS. 1 and 6).

In step 830, values of corresponding pixels of other captured images aresequentially added to the initial value generated in step 820, while thesum is less than the ceiling value received in step 810. Thus, in thecase of a bright image portion, the summation may be limited to a smallsubset of the available pixel values. Conversely, in the case of a darkimage portion, the summation may include all available pixel values.Corresponding pixels from other captured images are selected accordingto discussion in connection with step 320 of method 300 (FIG. 3), tomatch the progression of an object of interest through the scene. TDImodule 640 generates a TDI pixel value that corresponds to stopping thesummation before the sum exceeds the ceiling value. For example, TDImodule 640 (FIG. 6) sequentially adds to the initial value of the TDIpixel values of corresponding pixels of images 115 (FIGS. 1, 2, and 6)different from the image 115 used in step 820, while the TDI pixel valueis less than ceiling value 644 (FIG. 6).

In certain embodiments, step 830 includes an optional step 835 forcentering the pixel values used in the summation about a desired line inthe captured images, such as a line corresponding to a finish linelocation. Step 835 ensures that all pixels in the TDI image are based onoptimally centered input data. Step 835 may be incorporated into step830 in an iterative fashion. For example, TDI module 640 (FIG. 6) mayuse a total of only two pixel values in steps 820 and 830, where the twopixel values are extracted from the earliest captured images 115 (FIGS.1, 2, and 6). This corresponds to an object of interest, such as object135 (FIGS. 1 and 2) being at its leftmost position in images 115.However, the image of the finish line may be shifted from this positionby a number of pixels and step 830 repeats the summation using two linesof images 115 that are optimally centered about the finish line image.

In step 840, the TDI pixel value is scaled according to the number ofpixel values used in step 830. This maintains the original relativescale of dark and bright portions of the image. For example, TDI module640 (FIG. 6) multiplies the TDI pixel value generated in step 830 with avalue included in digital gain setting 646 (FIG. 6). This value may bethe factor N_(max)/N_(used), where N_(max) is the full number of pixelvalues available in step 830, and N_(used) is the number of pixel valuesused in step 830.

After performing steps 820, 830, and 840 for all pixels of the TDIimage, the TDI pixels are combined to form the TDI image in a step 850.For example, TDI module 640 (FIG. 6) combines all TDI pixel valuesgenerated by step 830 to form TDI image 145 (FIGS. 1 and 6).

In an optional step 860, TDI images generated in step 850 are normalizedto represent the images with a lower dynamic range. For example, TDImodule 640 (FIG. 6) applies standard image processing methods known to aperson ordinarily skilled in the art, such as gamma corrections, togenerate a normalized TDI image with a bit depth identical to that ofarea scan image sensor 610 used to capture images 115.

In certain embodiments, method 800 is executed such that the number oflines used to generate the TDI image is determined on an individual TDIline basis. These embodiments may advantageously be performed with step860 included in method 800. In these embodiments, the number ofcontributing pixel values in steps 830 and 840 is identical for all TDIpixels belonging to a given TDI line. Step 830 may be executed with“sum” being, for example, the maximum value of individual TDI pixel sumsassociated with the TDI line under consideration. In this case, thebrightest TDI pixel defines the number of contributing pixel values forthe TDI line. Alternatively, step 830 may be executed with “sum” beingthe average value of individual TDI pixel sums associated with the TDIline under consideration. In another example, step 830 is executed with“sum” being n'th percentile of individual TDI pixel sums associated withthe TDI line under consideration, where n is a number between 0 and 100.

FIG. 9 illustrates one exemplary method 900 for improving the dynamicrange of TDI images using fractional TDI. Method 900 is an embodiment ofmethod 800 of FIG. 8 further including fractional TDI. Fractional TDIallows for the inclusion of fractions of captured image pixel valuescontributing to a TDI pixel. For comparison, method 800 is restricted tointeger steps in the number of captured image pixel values contributingto a TDI pixel. Hence, two adjacent TDI pixels, associated with similarbrightness in the captured images and generated using method 800, mayhave, for example, two and three contributing pixel values,respectively. In some situations, a discreet step in noise and/or blurproperties between the two adjacent TDI pixels may result therefrom.Method 900, on the other hand, allows for adding fractional pixel valuessuch that the two adjacent TDI pixels discussed above show a smoothtransition in noise and/or blur properties.

Method 900 first performs step 810 (FIG. 8). Next, method 900 performssteps 820 (FIG. 8), 930, and 940 for all pixels in the TDI image. Step930 is a modification of step 830 (FIG. 8) further utilizing fractionalpixel values. In step 930, values of corresponding pixels of othercaptured images are sequentially added to the initial value generated instep 820, until the sum equals the ceiling value received in step 810.Generally, the last pixel value added is a fractional pixel value. Forexample, TDI module 640 (FIG. 6) sequentially adds to the initial valueof the TDI pixel, extracted from line 118(1,6) (FIG. 2), the value ofthe corresponding pixel of line 118(2,7) (FIG. 2) and a fraction x ofthe corresponding pixel value from line 118(3,8) (FIG. 2), where thefraction x is between zero and one. Thus, in this example, a total of2+x pixel values contribute to the TDI pixel value. In an embodiment,step 930 further includes step 835, as discussed in connection with FIG.8. Step 940 is a modification of step 840 (FIG. 8) that allows fornon-integer scaling of the TDI pixel value generated in step 930 toaccount for the inclusion of fraction pixel values in step 930. Forexample, TDI module 640 (FIG. 6) multiplies the TDI pixel valuegenerated in step 930 with a value included in digital gain setting 646(FIG. 6). This value may be the factor N_(max)/N_(used), where N_(max)is the full number of pixel values available in step 930, and N_(used)is the, possibly non-integer, number of pixel values used in step 930.Using the example discussed in connection with step 930, N_(used) is2+x. After completing steps 820, 930, and 940 for all pixel of the TDIimage, method 900 performs step 850 (FIG. 8) and optionally step 860(FIG. 8).

As discussed for method 800 of FIG. 8, method 900 may be executed suchthat the number of lines used to generate the TDI image is determined onan individual TDI line basis.

FIGS. 10A and 10B illustrate one exemplary filtered area scan imagesensor 1000 that includes an area scan image sensor 1010 and a positiondependent filter 1020. FIGS. 10A and 10B may sometimes be collectivelyreferred to herein as FIG. 10. Filtered scan image sensor is anembodiment of area scan image sensor 100 of FIG. 1 and of area scanimage sensor 610 of FIG. 6. FIG. 10A illustrates filtered area scanimage sensor 1000 in elevational view. FIG. 10B illustrates filteredarea scan image sensor 1000 in top plan view. Area scan image sensor1010 includes a photosensitive pixel array 1012 and, optionally, a colorfilter array 1014 for providing color information. Optional color filterarray 1014 is, for example, a Bayer type array.

Position dependent filter 1020 includes five spatially separated filterportions 1025(1), 1025(2), 1025(3), 1025(4), and 1025(5) for filteringlight propagating towards area scan image sensor 1010. Filter portion1025(1) is an infrared filter for at least partially blocking infraredlight. Filter portions 1025(2), 1025(3), and 1025(4) are neutral densityfilters with three different transmission coefficients. Filter portion1025(5) is a blank filter for transmitting substantially all incidentlight. In one embodiment, position dependent filter 1020 is fixed toarea scan image sensor 1010. Filter 1020 is, for example, applied toarea scan image sensor 1010 using one or more coating methods known inthe art. In another embodiment, position dependent filter 1020 ismounted close to area scan image sensor 1010 and fixed in relationthereto. The brightness of an object 135 (FIG. 1), as captured byfiltered area scan image sensor 1000, depends on the position of object135 in the image. For example, object 135 will appear brighter in aportion of the image associated with blank filter portion 1025(5) thanin a portion of the image associated with neutral density filter1025(2).

Filtered area scan image sensor 1000 may include more or fewer filterportions 1025 than illustrated in FIG. 10 without departing from thescope hereof. Additionally, filtered area scan image sensor 1000 mayinclude other types of brightness adjusting filter portions than thoseillustrated in FIG. 10 without departing from the scope hereof.

FIG. 11 illustrates one exemplary method 1100 for processing eventtiming images to adjust the brightness of a TDI image by using an areascan image sensor with a position-dependent filter. Method 1100 may beperformed, for example, by system 100 (FIG. 1) with filtered area scanimage sensor 1000 (FIG. 10) implemented as area scan image sensor 110(FIG. 1), or by system 600 with area scan image sensor 1000 (FIG. 10)implemented as area scan image sensor 610 (FIG. 1).

In a step 1110, a portion of the two-dimensional images, captured by anarea scan image sensor with a position-dependent filter, is selected.The portion is associated with a certain filter portion. Step 1110 mayserve to adjust the brightness of a TDI image generated therefrom. Forexample, in system 600 (FIG. 6) with filtered area scan image sensor1000 (FIG. 10) implemented as area scan image sensor 610, TDI module 640selects a spatial portion of images 115 (FIGS. 1 and 6) associated witha certain filter portion 1025 (FIG. 10) to achieve a desired brightnessof TDI image 145 (FIGS. 1 and 6).

In an optional step 1120, the alignment of a camera that houses the scanimage sensor with a position dependent filter is adjusted. This isrelevant in a use scenario where the camera has been aligned such that,for example, a finish line is imaged onto a particular line of pixels ofthe image sensor. The finish line may not be imaged onto a portion ofthe sensor associated with the selection made in step 1110. Optionalstep 1120 is performed, for example, by system 600 (FIG. 6), withfiltered area scan image sensor 1000 (FIG. 10), implemented into TDIcamera system 2000 (FIG. 20) as TDI camera 2010. Using method 2100 (FIG.21) and/or method 2300 (FIG. 23), alignment control system 2040 (FIG.20) realigns camera 670 (FIG. 6) such that a finish line is imaged ontoa portion of filtered area scan image sensor 1000 associated with thespatial portion of images 115 selected in step 1110. In an optional step1130, area scan images area captured with the new camera alignmentachieved in step 1120. For example, area scan image sensor 610 (FIG. 6)captures two-dimensional images 115 (FIGS. 1 and 6).

Next, method 1100 proceeds to perform step 830, and optionally step 840,of method 800 (FIG. 8). If performing optional step 840, method 1100 mayreturn to step 1110 for further adjustment.

Optional steps 1120 and 1130 are performed, for example, in a scenariowhere step 1110 is performed prior to the occurrence of an event ofinterest, such as the finish of a race. In this exemplary scenario, step1110 and optional step 1120 may be performed during setup of an eventtiming system, while optional step 1130 is performed during the event.In another exemplary scenario, method 1100 is processing event timingimages during an event such as the finish of a race. In this scenario,steps 1110, 830, and optionally step 840, are performed while raceparticipants cross a finish line. If time allows, for example if thereis a sufficient time gap between two subsequent race finishers, thisexample may include performing optional steps 1120 and 1130 during theinterim between the two subsequent race finishers crossing the finishline.

All of methods 800 (FIG. 8), 900 (FIG. 9), and 1100 (FIG. 11) areconcerned with achieving a certain brightness of a TDI image. Two ormore of these methods may be performed in conjunction, or one or more ofthe methods may be performed separately, to generate a TDI image.

FIG. 12 illustrates one exemplary method 1200 for processing eventtiming images captured by a color area scan image sensor having a Bayertype pixel array. Method 1200 generates a color TDI image with twice theresolution of the color area scan image sensor. Accordingly, the TDIimage generated by method 1200 provides twice the time resolution ascompared to the TDI images generated by method 500 (FIG. 5). Method 1200is applicable, for example, to the generation of TDI images by system100 (FIG. 1) with color area scan image sensor 400 (FIG. 4) implementedas area scan image sensor 110 (FIG. 1). As discussed in connection withFIG. 4, photosites 421, 422, 423, and 424 may be arranged differentlywithin color pixel 420. For example, the locations of two or more ofphotosites 421, 422, 423, and 424 may be swapped as compared to theillustration of FIG. 4. Method 1200 may be correspondingly modified fromthe embodiment illustrated in FIG. 12 to apply to such alternate imagesensor layouts, without departing from the scope hereof. In terms ofresolution, each line 118 (FIG. 2) of a captured two-dimensional image115 (FIGS. 1 and 2) corresponds to two neighboring lines of the TDIimage. Method 1200 assumes that the images 115 (FIGS. 1 and 2) arecaptured by the color area scan image sensor at a frame rate such thatan object of interest progresses through lines 118 (FIG. 2) of the colorarea scan image sensor at a rate of one line per frame. For example,color area scan image sensor 400 (FIG. 4) captures images of scene 130(FIG. 1) at a rate such that object 135 (FIG. 1) progresses throughlines 410 (FIG. 4) at a rate of one line per frame. Method 1200processes such images and is performed, for example, by TDI module 140(FIG. 1).

In a step 1210, each two-dimensional image, captured by the color areascan image sensor, is received in the form of rows. The rows areoriented parallel with the lines of method 300 (FIG. 3), such that aline of method 300 corresponds to two rows of method 500. The two rowsare an R&G row composed of signals from R and G photosites and a G′&Brow composed of signals from G′ and B photosites. In one embodiment, theimages are received from a stored location. In another embodiment, theimages are received from the area scan image sensor used to capture theimages. For example, TDI module 140 (FIG. 1) receives two-dimensionalimages 115 (FIG. 1) captured by color area scan image sensor 400 (FIG.4) as rows, such that each line 410 (FIG. 4) is associated with tworows: (a) a row composed of all R1 (421) and G1 (422) photosite signalsfrom line 410 and (b) a row composed of all G1′ (423) and B1 (424)photosite signals from line 410. In another example, TDI module 140(FIG. 1) receives two-dimensional images 115 (FIG. 1), captured by colorarea scan image sensor 400 (FIG. 4), in any arbitrary format. TDI module140 (FIG. 1) processes the two-dimensional images 115 (FIG. 1) togenerate rows, such that each line 410 (FIG. 4) is associated with tworows: (a) a row composed of all R1 (421) and G1 (422) photosite signalsfrom line 410 and (b) a row composed of all G1′ (423) and B1 (424)photosite signals from line 410.

The TDI image, generated by method 1200, is composed of lines zerothrough N, where N is an odd integer. The lines of the TDI imagegenerated by method 1200 are equivalent to lines 146 of FIG. 2, exceptthat each line 146 of FIG. 2 corresponds to two lines of the TDI imagegenerated by method 1200. Following step 1210, method 1200 performssteps 1220, 1230, 1241, 1242, 1251, 1252, 1261, and 1262 for each pairof neighboring even and odd TDI lines.

In step 1220, a series of R&G rows, each from a different image, isformed. The series of R&G rows follows the progression of an objectthrough a scene, as discussed in connection with FIGS. 2 and 3. Forexample, TDI module 140 (FIG. 1) forms a series of R&G rows associatedwith the respective series of lines 410(1), 410(2), and 410(3) of colorarea scan image sensor 400 (FIG. 4). The series of R&G rows areextracted from a respective series of sequentially captured images 115(FIG. 1). In step 1230, the series of R&G rows generated in step 1220 isintegrated to form a single, integrated R&G row. For example, TDI module140 (FIG. 1) integrates the series of R&G rows generated in step 1220 toform a single, integrated R&G row. Step 1230 may performed at any timeafter step 1220 and before steps 1261 and 1262. Method 1200 proceeds toperform sequential steps 1241, 1251, and 1261 to populate the even TDIline and steps 1242, 1252, and 1262 to populate the odd TDI line.Sequential steps 1241, 1251, and 1261 may be performed in series orparallel with sequential steps 1242, 1252, and 1262.

In step 1241, a series of G′&B rows are formed. The series of G′&B rowsis matched to the series of R&G rows formed in step 1220, such that eachG′&B row from the series of G′&B rows is extracted from the same line ofthe same captured image as a respective one of the series of R&G rowsgenerated in step 1220. For example, TDI module 140 (FIG. 1) forms aseries of G′&B rows associated with the respective series of lines410(1), 410(2), and 410(3) of color area scan image sensor 400 (FIG. 4),where the series of R&G rows formed in step 1220 is also associated withthe respective series of lines 410(1), 410(2), and 410(3).

In step 1251, the matched series of G′&B rows generated in step 1241 isintegrated to form a single, matched integrated G′&B row. For example,TDI module 140 (FIG. 1) integrates the matched series of G′&B rows toform a single, matched integrated G′&B row.

In step 1261, the integrated R&G row generated in step 1230 and thematched integrated G′&B row generated in step 1251 are combined to forma single color line including at least R, G″, and B data for each pixel.The even TDI line is populated with this single color line. In oneembodiment, the R, G″, and B data of each pixel of the single color lineincludes (a) the R data from the corresponding integrated R&G row, (b)the average of the G and G′ data, to form the G″data, from thecorresponding pixels of the integrated R&G row and the matchedintegrated G′&B row, respectively, and (c) the B data of the from thecorresponding integrated G′&B row. In another embodiment, both G datafrom the integrated R&G row and G′ data from the matched integrated G′&Brow are retained. In this embodiment, the R, G″, and B data of eachpixel of the single color line includes (a) the R data of the from thecorresponding integrated R&G row, (b) the G data from the correspondingpixel of the integrated R&G row, (c) the G′ data from the correspondingpixel of the integrated G′&B row, and (d) the B data of the from thecorresponding integrated G′&B row. In this embodiment, the G″ dataincludes the G data and the G′ data. For example, TDI module 140(FIG. 1) combines the integrated R&G row generated in step 1130 with thematched integrated G′&B row generated in step 1251 to populate the evenTDI line.

In step 1242, a series of G′&B rows are formed. The series of G′&B rowsis shifted in time by one image frame relative to the series of R&G rowsformed in step 1220. Hence, each G′&B row from the series of G′&B rowsis extracted from the same line position as the corresponding R&G row,but from an image that is one frame earlier than the image from whichthe corresponding R&G row is extracted. For example, TDI module 140(FIG. 1) forms a series of G′&B rows associated with the respectiveseries of lines 410(1), 410(2), and 410(3) of color area scan imagesensor 400 (FIG. 4), where the series of R&G rows formed in step 1220 isalso associated with the respective series of lines 410(1), 410(2), and410(3). However, the series of G′&B rows is extracted from a respectiveseries of sequentially captured images that is shifted in time by oneframe, as compared to the series of sequentially captured images used instep 1220.

In step 1252, the shifted series of G′&B rows generated in step 1242 isintegrated to form a single, shifted integrated G′&B row. For example,TDI module 140 (FIG. 1) integrates the shifted series of G′&B rows toform a single, shifted integrated G′&B row.

In step 1262, the integrated R&G row generated in step 1230 and theshifted integrated G′&B row generated in step 1252 are combined to forma single color line including at least R, G″, and B data for each pixel.The odd TDI line is populated with this single color line. Thus, the oddTDI line is composed of “crossover” color pixels, as each pixel of theodd TDI line is generated from sets of photosites extracted fromdifferent image frames. In one embodiment, the R, G″, and B data of eachpixel of the single color line includes (a) the R data of the from thecorresponding integrated R&G row, (b) the average of the G and G′ datafrom the corresponding pixels of the integrated R&G row and the matchedintegrated G′&B row, respectively, and (c) the B data of the from thecorresponding integrated G′&B row. In another embodiment, both G datafrom the integrated R&G row and G′ data from the matched integrated G′&Brow are retained. In this embodiment, the R, G″, and B data of eachpixel of the single color line includes (a) the R data of the from thecorresponding integrated R&G row, (b) the G data from the correspondingpixel of the integrated R&G row, (c) the G′ data from the correspondingpixel of the integrated G′&B row, and (d) the B data of the from thecorresponding integrated G′&B row. For example, TDI module 140 (FIG. 1)combines the integrated R&G row generated in step 1230 with the shiftedintegrated G′&B row generated in step 1252 to populate the odd TDI line.

Following steps 1241 and 1242, method 1200 proceeds to perform step 360of method 300 (FIG. 3).

Referring to the direction of object movement 430 in FIG. 4, the objectmoves from R and G photosites 421(1) and 422(2), respectively, to G′ andB photosites 423(1) and 424(1), respectively, as time progresses. Theeven TDI lines are matched to the object position when the object iscentered on a given line 410(i). The odd TDI lines are matched to theobject position when the object is centered on the dividing line betweenthe line 410(i) and the line 410(i+1). Accordingly, method 1200 utilizesthe individual photosites of Bayer type color area scan image sensor 400(FIG. 4) to generate a TDI image with double resolution.

Method 1200 may be extended to other orientations of Bayer-typephotosite layout without departing from the scope hereof. For example,method 1200 may be extended to a Bayer-type photosite layout rotated byninety degrees as compared to the layout illustrated in FIG. 4. Method1200 may also be extended to non-Bayer type color area scan sensorswithout departing from the scope hereof. For example, method 1200 may beutilized to process images captured by color area scan image sensors,wherein each color pixel is composed of four unique photosites. Suchcolor area scan image sensors include color area scan image sensorsconfigured with an RGBE (red, green, blue, emerald) or a CYGM (cyan,yellow, green, magenta) color filter array.

In another example, method 1200 is extended to process images capturedby trilinear color image sensor 1500 of FIG. 15. In this case, the twoparallel sets of sequential steps (steps 1241, 1251, and 1261, and steps1242, 1252, and 1262) are replaced by three equivalent parallel sets ofsequential steps: (a) a set of steps processing R, G, and B lines from aseries of sequentially captured color pixel lines following theprogression of an object through the image frame, (b) a set of stepsprocessing R and G lines from one series of sequentially captured colorpixel lines, following the progression of an object through the imageframe, with B lines from another series shifted therefrom in time by oneimage frame, and (c) a set of steps processing R lines from one seriesof sequentially captured color pixel lines, following the progression ofan object through the image frame, with G and B lines from anotherseries of color pixel lines shifted therefrom in time by one imageframe. Accordingly, method 1200 generates TDI images with tripletemporal resolution as compared to TDI images generated using method 300(FIG. 3).

FIG. 13 illustrates one exemplary method 1300 for generating a TDI imagefrom images captured by a color area scan image sensor having a Bayertype pixel array. Method 1300 generates TDI images showing an object ofinterest with twice the resolution, as compared to the TDI imagesgenerated by method 500 (FIG. 5). Method 1300 utilizes images capturedat twice the frame rate, as compared to the images processed in method500 (FIG. 5), as well as processing of individual photosites. In thecontext of an event timing system, the TDI image generated by method1300 provides twice the time resolution as compared to the TDI imagesgenerated by method 500 (FIG. 5). Method 1300 is applicable, forexample, to the generation of TDI images by system 100 (FIG. 1) withcolor area scan image sensor 400 (FIG. 4) implemented as area scan imagesensor 110 (FIG. 1). In the present discussion of method 1300, colorarea scan image sensor is a Bayer type image sensor. However, method1300 may be extended to generating a TDI image using images captured byany color area image sensor where each color pixel is composed of atwo-by-two photosite array, without departing from the scope hereof.

Method 1300 assumes that images 115 (FIG. 2) are captured by the colorarea scan image sensor at a frame rate such that an object of interestprogresses through lines 118 (FIG. 2) of the color area scan imagesensor at a rate of half a line per frame. The image processing ofmethod 1300 is similar to method 500 (FIG. 5) except that the imageprocessing of method 1300 accounts for the images being captured attwice the frame rate. Method 1300 is performed, for example, by system100 (FIG. 1) with color area scan image sensor 400 (FIG. 4) implementedas area scan image sensor 110 (FIG. 1).

In an optional step 1310, a color area scan image sensor capturessequential images 0 through N, where N is an odd integer, of an objectpassing through a scene. Hence, the image series is composed ofalternating even and odd number images. The color area scan image sensorcaptures images at a frame rate such that an object of interestprogresses through the frame at a rate of half a line per frame. Forexample, color area scan image sensor 400 (FIG. 4) captures images 115(FIG. 1) at a frame rate such that object 135 (FIG. 1) progressesthrough scene 130 (FIG. 1) at a rate of half a line 118 (FIG. 2) perframe.

In a step 1320, each two-dimensional image captured by the color areascan image sensor is received in the form of R&G rows and G′&B rows, asdefined above. In an embodiment of method 1200 that includes optionalstep 1310, step 1320 receives the images captured in step 1310. In anembodiment of method 1300 that does not include optional step 1310, theimages may be received from elsewhere, for example from a storedlocation. The rows are oriented parallel with the lines of method 300(FIG. 3) such that a line of method 300 corresponds to two rows ofmethod 1300: an R&G row composed of all R and G photosite signals, and aG′&B row composed of all G′ and B photosite signals. Accordingly, anobject of interest passes through the sequentially captured images at arate of half a line per image frame. For example, TDI module 140(FIG. 1) receives two-dimensional images 115 (FIG. 1) captured by colorarea scan image sensor 400 (FIG. 4) as rows, such that each line 410(FIG. 4) is associated with two rows: (a) a row composed of all R (421)and G (422) photosite signals from line 410 and (b) a row composed ofall G′ (423) and B (424) photosite signals from line 410. In anotherexample, TDI module 140 (FIG. 1) receives two-dimensional images 115(FIG. 1), captured by color area scan image sensor 400 (FIG. 4) in anarbitrary format. TDI module 140 (FIG. 1) processes the two-dimensionalimages 115 (FIG. 1) to generate rows, such that each line 410 (FIG. 4)is associated with two rows: (a) a row composed of all R and G photositesignals from line 410 and (b) a row composed of all G′ and B photositesignals from line 410.

Following step 1320, method 1300 proceeds to populate each line of theTDI image by performing steps 1331, 1332, 531 (FIG. 5), 532 (FIG. 5),and 540 (FIG. 5) for each line in the TDI image. Steps 1331 and 531 areperformed sequentially, as are steps 1332 and 532. Sequential steps 1331and 531 may be performed in parallel or series with sequential steps1332 and 532.

In step 1331, a series of R&G rows, each from a different image, isformed. The series of R&G rows follows the progression of an objectthrough a scene. The series of R&G rows is extracted from at least aportion of the even-numbered images. For example, TDI module 140(FIG. 1) forms a series of R&G rows associated with the respectiveseries of lines 410(1), 410(2), and 410(3) of color area scan imagesensor 400 (FIG. 4). The series of R&G rows are extracted from arespective series of sequentially captured even-numbered images 115(FIG. 1). After performing step 1331, method 1300 proceeds to performstep 531 of method 500 (FIG. 5).

In step 1332, a series of G′&B rows, each from a different image, isformed. The series of G′&B rows follows the progression of an objectthrough a scene. The series of G′&B rows is extracted from at least aportion of the odd-numbered images. For example, TDI module 140 (FIG. 1)forms a series of G′&B rows associated with the respective series oflines 410(1), 410(2), and 410(3) of color area scan image sensor 400(FIG. 4). The series of G′&B rows are extracted from a respective seriesof sequentially captured odd-numbered images 115 (FIG. 1). Afterperforming step 1332, method 1200 proceeds to perform step 532 of method500 (FIG. 5).

Following the performance of steps 531 and 532, method 1300 proceeds toperform step 540 of method 500 (FIG. 5) and step 360 of method 300 (FIG.3).

Method 1300 may be extended to other orientations of Bayer-typephotosite layout without departing from the scope hereof. For example,method 1300 may be extended to a Bayer-type photosite layout rotated byninety degrees as compared to the layout illustrated in FIG. 4. Method1300 may also be extended to non-Bayer type color area scan sensorswithout departing from the scope hereof. For example, method 1200 may beutilized to process images captured by color area scan image sensors,wherein each color pixel is composed of four unique photosites. Suchcolor area scan image sensors include color area scan image sensorsconfigured with an RGBE (red, green, blue, emerald) or a CYGM (cyan,yellow, green, magenta) color filter array.

In another example, method 1300 is extended to process images capturedby trilinear color image sensor 1500 of FIG. 15. In this case, imagesare captured at triple frame rate. The two parallel sets of sequentialsteps (steps 1331 and 531, and steps 1332 and 532) are replaced by threeequivalent parallel sets of sequential steps: (a) a set of stepsprocessing R lines from a series of sequentially captured color pixelslines following the progression of an object through the image frame,(b) a set of steps processing G lines from a series of sequentiallycaptured color pixels lines following the progression of an objectthrough the image frame, and (c) a set of steps processing B lines froma series of sequentially captured color pixels lines following theprogression of an object through the image frame. Accordingly, the TDIimage generated by method 1300 has temporal resolution triple that ofTDI images processed according to method 300 (FIG. 3).

FIG. 14 illustrates two adjacent image lines produced by a portion ofone exemplary Bayer type color area scan image sensor 1400, whereindividual photosites are used to double the spatial resolution of acamera. Thus, color area scan image sensor 1400 is advantageouslyimplemented in an image based event timing system. Color area scan imagesensor 1400 includes a Bayer type pixel array. Each color pixel 1420 ofan image generated by color area scan image sensor 1400 is composed offour photosite signals 1421, 1422, 1423, and 1424. In an embodiment,photosite signal 1421 represents red (R) light, photosite signals 1422and 1423 represent green (G) light, and photosite signal 1424 representsblue (B) light. For a pair of adjacent image lines 1410(1) and 1410(2),three output image lines are generated. Two of the output image linesare simply the original image lines 1410(1) and 1410(2), each pixelthereof being composed of photosite signals R1 (1421(1)), G1 (1422(1)),G1′ (1423(1)), and B1 (1424(1)) for image line 1410(1), and R2(1421(2)), G2 (1422(2)), G2′ (1423(2)), and B2 (1424(2)) for image line1410(2). The third output image line is generated as crossover colorpixels 1430, each composed of a combination of photosite signals fromthe original image lines 1410(1) and 1410(2), specifically photositesignals G1′ (1423(1)) and B1 (1424(1)) of image line 1410(1) andphotosites R2 (1421(2)) and G2 (1422(2)) of image line 1410(2).

An image composed of crossover color pixels in addition to originalcolor pixels provides color pixel lines at twice the spatial resolutionas compared to a color image generated without the use of crossovercolor pixels, as every pair of adjacent original image lines may be usedto form a third image line composed of crossover color pixels. A seriesof such images, captured sequentially, may be processed to form a TDIimage with lines at twice the resolution as compared to a TDI imagebased on conventional color images without crossover pixels.

The lines generated from color area scan image sensor 1400 may be usedas input to TDI processing of images as discussed in connection withFIGS. 1, 2, and 3. In an embodiment, color area scan image sensor 1400is implemented as area scan image sensor 110 in system 100. TDI module140 processes standard color images to (a) generate higher resolutionimages composed of original color pixels and crossover color pixels and(b) form a TDI image, according to method 300 (FIG. 3), using bothcrossover pixel lines and original pixel lines. For system 100, used forexample as a photo finish camera, where images are used to time an eventor separate two or more events in time, this results in a doubling ofthe time resolution over that provided by the original images.

For comparison, the crossover color pixels of method 1200 (FIG. 12)result from a temporal cross-over of photosites performed whilecombining rows from different images to form the TDI image. Thecrossover color pixels of FIG. 14 result from a spatial cross-over ofphotosites in the originally captured images.

FIG. 15 illustrates two adjacent image lines produced by a portion ofone exemplary trilinear color image sensor 1500, where individual linesof photosites are used to triple the spatial resolution of a camera.Accordingly, trilinear color image sensor 1500 is advantageouslyimplemented in an image based event timing system. In one embodiment,trilinear color image sensor 1500 is a line scan image sensor with asingle set of photosite lines to form a single line of color pixels. Inanother embodiment, trilinear color image sensor 1500 is an area scanimage sensor with a plurality of sets of photosite lines forming acorresponding plurality of color pixel lines. This embodiment oftrilinear color image sensor 1500 may be implemented in system 100 asarea scan image sensor 110. For both of these two embodiments, eachcolor pixel line of trilinear color image sensor 1500 is composed ofthree lines of photosites, each line of photosites having a differentcolor sensitivity. In an embodiment, a color pixel line 1510 of an imagegenerated by trilinear color image sensor 1500 is composed of signalsfrom three photosite lines such that each color pixel 1520 of colorpixel line 1510 is composed of three photosite signals 1521, 1522, and1523 representative of red, green, and blue light, respectively.

For a pair of adjacent image lines 1510(1) and 1510(2), three outputimage lines are generated. Two of the output image lines are theoriginal line image frames 1510(1) and 1510(2), each pixel thereof beingcomposed of photosite signals R1 (1521(1)), G1 (1522(1)), and B1(1523(1)) for image line 1510(1), and R2 (1521(2)), G2 (1522(2)), and B2(1523(2)) for image line 1510(2). A third output image line is generatedas crossover color pixels 1531, each composed of a combination ofphotosites from the original adjacent image lines 1510(1) and 1510(2),specifically photosite signals G1 (1522(1)) and B1 (1523(1)) of imageline 1510(1) and photosite signal R2 (1521(2)) of image line 1510(2).Similarly, a fourth output image line is generated as crossover colorpixels 1532 composed of combination photosites from the originaladjacent image lines 1510(1) and 1510(2), specifically photosite signalB1 (1523(1)) of image line 1510(1) and photosite signal R2 (1521(2)) andG2 (1522(2)) of image line 1510(2). An image composed of crossover colorpixels provides color pixel lines at three times the spatial resolutionof the original color images, as every pair of adjacent original imagelines may be used to form two additional line image frames composed ofcrossover pixels.

As discussed for FIG. 14, the image lines generated from trilinear colorimage sensor 1500 may be used as input to TDI processing. In anembodiment, trilinear color image sensor 1500 is implemented as areascan image sensor 110 in system 100. TDI module 140 processes standardcolor images to (a) generate higher resolution images composed oforiginal color pixels and crossover color pixels and (b) form a TDIimage, according to method 300 (FIG. 3), using both crossover pixellines and original pixel lines. For system 100, used for example as aphoto finish camera, where images are used to time an event or separatetwo or more events in time, this results in a tripling of the timeresolution over that provided by the original images.

For comparison, the crossover color pixels of method 1200 (FIG. 12)result from a temporal cross-over of photosites performed whilecombining rows from different images to form the TDI image. Thecrossover color pixels of FIG. 15 result from a spatial cross-over ofphotosites in the originally captured images.

FIG. 16 illustrates one exemplary method 1600 for capturing andprocessing event timing images. Method 1600 may be executed by system100 of FIG. 1 or system 600 of FIG. 6, for example. In a step 1610,two-dimensional images are captured by an area scan image sensor, forexample area scan image sensor 110 of FIG. 1, color area scan imagesensor 400 of FIG. 4, filtered area scan image sensor 1000 of FIG. 10,color area scan image sensor 1400 of FIG. 14, trilinear color imagesensor 1500 of FIG. 15, or diagonal CFA area scan image sensor 1700(FIG. 17). In a step 1620, the captured images are communicated to anexternal image processing module, for example TDI module 140 of FIG. 1.In a step 1630, the external image processing module performs TDI of thecaptured images and/or the high resolution images generated in optionalstep 1630. If the captured images are color images captured by a colorsensor such as color area scan image sensor 400 (FIG. 4), color areascan image sensor 1300 (FIG. 13), trilinear color image sensor 1400(FIG. 14), or diagonal CFA area scan image sensor 1700 (FIG. 17), step1630 may include a step 1640, wherein higher resolution images may begenerated using crossover color pixels. This is discussed for temporalcross-over in connection with method 1200 (FIG. 12), and for spatialcross-over in connection with FIGS. 14 and 15. TDI may be performed byTDI module 140 (FIG. 1) using method 300 of FIG. 3, method 500 of FIG.5, method 1200 of FIG. 12, or method 1300 of FIG. 13. A step 1650outputs the TDI image Step 1650 may be performed by interface 150 ofFIG. 1.

In one embodiment, the external image processing module outputs the TDIimage when complete. In another embodiment, the external imageprocessing module outputs the TDI image one pixel, row, or line at atime, in the manner that the pixels, rows, or lines are generated by theexternal image processing module.

In certain embodiments, steps 1610 and 1620 are omitted. A dataprocessing system, such as TDI module 140 (FIG. 1) or a computer withimage processing capability, receives images captured by an area scanimage sensor and performs steps 1630 and 1650.

FIG. 17 illustrates one exemplary diagonal color filter array (CFA) areascan image sensor 1700, wherein each color pixel includes a 3×3photosite array. Diagonal CFA area scan image sensor 1700 offersenhanced image processing flexibility and extends the highone-dimensional resolution provided by trilinear color image sensor 1500(FIG. 15) to two dimensions. Diagonal CFA area scan image sensor 1700 iscomposed of lines 1710(i), where i is a positive integer. Threeexemplary adjacent lines 1710(1), 1710(2), and 1710(3) are shown in FIG.17, although diagonal CFA image sensor 1700 may have any number of lines1710, without departing from the scope hereof. Each line 1710(i) iscomposed of color pixels 1720(i,j), where j indicates the verticalposition, as oriented in FIG. 17, of color pixel 1720(i,j) within line1710(i). FIG. 17 shows three exemplary color pixels: color pixel1720(1,1) in line 1710(1), color pixel 1720(1,2) in line 1710(2), andcolor pixels 1720(1,3), 1720(2,3), and 1720(3,3) in line 1710(3). Eachline 1710(i) may include any number of color pixels 1720(i,j). Incertain embodiments, all lines 1710 include the same number of colorpixels 1720.

Each color pixel 1720(i,j) includes a 3×3 array of photosites1721(i,j)(n,m), where n and m are positive integers smaller than orequal to three. Not all photosites 1721 are explicitly numbered in FIG.17. Each color pixel 1720(i,j) includes photosites 1721(i,j)(1,1),1721(i,j)(2,3), and 1721(i,j)(3,2), sensitive to a first color,photosites 1721(i,j)(1,2), 1721(i,j)(2,1), and 1721(i,j)(3,3), sensitiveto a second color, and photosites 1721(i,j)(1,3), 1721(i,j)(2,2), and1721(i,j)(3,1), sensitive to a third color. In an embodiment, the first,second, and third colors are red (R), green (G), and blue (B). However,diagonal CFA image sensor 1700 may be implemented using other colorsensitivity configurations, such as cyan, magenta, and yellow, withoutdeparting from the scope hereof. According to the illustration in FIG.17, photosites of same color sensitivity form diagonal lines. Colorpixels 1720 may be oriented differently, for example such that thediagonal lines formed by photosites of same color sensitivity arerotated by ninety degrees, as compared to FIG. 17, without departingfrom the scope hereof. Photosites 1721 are arranged such that any columnof three photosites includes a first-color photosite, a second-colorphotosite, and a third-color photosite, and any row of three photositesincludes a first-color photosite, a second-color photosite, and athird-color photosite, wherein the first-, second, and third-colorphotosites are sensitive to light of first, second, and third color,respectively. This arrangement offers increased flexibility for groupingof photosites 1721 during processing of photosite signals generated byphotosites 1721 of diagonal CFA area scan image sensor 1700.

In one use scenario, images captured by diagonal CFA image sensor 1700are processed retaining individual color pixels 1720 as separate itemsthroughout processing. In this scenario, images are processed accordingto, for example, method 300 (FIG. 3). Method 500 of FIG. 5 is extendableto processing of images captured by diagonal CFA image sensor 1700, forexample as hereinafter discussed. Step 510 is extended to receive threerows: an R&G′&B″ row, a G&B′&R″ row, and a B&R′&G″ row. Parallelprocesses including steps 521 and 531 and steps 522 and 532 are extendedto include three equivalent parallel processes operating on the R&G′&B″,G&B′&R″, and B&R′&G″ rows, respectively. Step 540 is extended to combinedata from three integrated rows.

In another use scenario, images captured by diagonal CFA image sensor1700 are processed using 2×2 photosite groups. Each 2×2 photosite groupsmay be fully within a single color pixel 1720 or include photosites fromtwo, three, or four adjacent color pixels 1720. In the latter case, a“full coverage set” of 2×2 photosite groups may be selected such thatthe set spans all photosites of diagonal CFA image sensor 1700, or acontiguous portion thereof. An exemplary 2×2 photosite group isindicated in FIG. 17 as 2×2 crossover color pixel 1740. This isequivalent to Bayer type color area scan image sensor 400 (FIG. 4)except that the 2×2 photosite groups of diagonal CFA image sensor 1700do not all have the same photosite layout. However, all 2×2 photositegroups include three different photosite types and therefore provide ascomplete color information as the color pixels of a Bayer type imagesensor, such as color pixels 420 of Bayer type color area scan imagesensor 400 (FIG. 4). In this use scenario, images captured by diagonalCFA image sensor 1700 are processed according to one or more of methods300 (FIG. 3), 500 (FIG. 5), 1200 (FIG. 12), 1300 (FIG. 13), or 1600(FIG. 16). When processing images captured by diagonal CFA image sensor1700 according to methods 1300 (FIG. 13) or 1600 (FIG. 16), a fullcoverage set of 2×2 photosite groups may be utilized to produce TDIimages with twice the resolution of captured images segmented into 2×2photosite groups. When processing images according to methods 500 (FIG.5), 1300 (FIG. 13), or 1600 (FIG. 16), processing is adapted to accountfor the fact that not all 2×2 photosite groups have the same photositelayout. Further, in the case of method 1300 (FIG. 13) applied todiagonal CFA image sensor 1700, images are captured at triple framerate, such that an object of interest moves at a rate of a third of aline 1710 per frame.

In yet another use scenario, images captured by diagonal CFA imagesensor 1700 are processed using color pixels 1720 as well as horizontalcrossover color pixels composed of 3×3 photosite arrays spanningportions of two adjacent color pixels 1720 located at the same verticalposition in FIG. 17. Horizontal crossover pixels are processed, forexample, in the same way as images captured by trilinear color imagesensor, according to methods 1200 (FIG. 12), 1300 (FIG. 13), and 1600(FIG. 16).

Diagonal CFA area scan image sensor 1700 has utility for generation ofTDI images, but may also be advantageous for use in other applicationstypically performed by line-scan cameras.

FIG. 18 illustrates one exemplary method 1800 for processing eventtiming images captured by a color area scan image sensor having colorpixels with two-dimensional photosite variation. Examples of color areascan images sensors having two-dimensional photosite variation includeBayer type color area scan image sensor 400 (FIG. 4) and diagonal CFAarea scan image sensor 1700 (FIG. 17). Method 1800 generates TDI imageswith improved resolution in the dimension parallel to the TDI lines.Optionally, TDI is performed at improved resolution such that the TDIimage has improved resolution in two dimensions. Method 1800 isperformed, for example, by TDI module 140 (FIG. 1).

In a step 1810, method 1800 receives images captured by a color areascan image sensor having color pixels with two-dimensional photositevariation. For example, TDI module 140 receives images captured bydiagonal CFA image sensor 1700 (FIG. 17) implemented as area scan imagesensor 110 (FIG. 1). After performing step 1810, method 1800 performsstep 1630 (FIG. 16) for two or more photosite group divisions of thecaptured images, where the two or more photosite group divisions aremutually shifted in the dimension parallel to the TDI lines.

Referring to FIG. 17, an assumed direction of motion for an object ofinterest is indicated by arrow 1760. Hence, TDI lines are orthogonal toarrow 1760 and parallel with lines 1710. For images captured by diagonalCFA image sensor 1700, step 1630 is performed for (a) a photosite groupdivision aligned, in the dimension parallel with lines 1710, with colorpixels 1720, (b) a photosite group division shifted from color pixels1720 by one photosite row, in the dimension parallel with lines 1710(for example aligned with photosite group 1750), and (c) a photositegroup division shifted from color pixels 1720 by two photosite rows, inthe dimension parallel with lines 1710.

Referring to FIG. 4, an assumed direction of motion for an object ofinterest is indicated by arrow 430. Hence, TDI lines are orthogonal toarrow 430 and parallel with lines 410. For images captured by Bayer typecolor area scan image sensor 400, step 1630 is performed for (a) aphotosite group division aligned, in the dimension parallel with lines410, with color pixels 420, and (b) a photosite group division shiftedfrom color pixels 420 by one photosite row, in the dimension parallelwith lines 410.

The multiple iterations of step 1630 generate respective TDI images withits respective color pixels centered on mutually shifted locations, inthe dimension parallel with the TDI lines. Optionally, step 1630includes step 1640 such that the TDI image has improved resolution inthe dimension orthogonal to the TDI lines. In an embodiment notillustrated in FIG. 18, step 1630 is replaced by method 1300 (FIG. 13),which also provides improved resolution in the dimension orthogonal tothe TDI lines.

In a step 1850, the TDI images generated in the multiple iterations ofstep 1630 are combined to form a TDI image with improved resolution inthe dimension parallel with the TDI lines. For example, TDI module 140combines TDI images generated in step 1630. This may be done using thesame method as discussed in connection with FIG. 14, however in theorthogonal dimension. After performing step 1850, method 1800 performsstep 1650 (FIG. 16).

Accordingly, method 1800 is capable of utilizing the individualphotosite data to maximize resolution in both dimensions. In the case ofimages captured by Bayer type color area scan image sensor 400 (FIG. 4),the resolution may be doubled in both dimensions, as compared to thecolor pixel resolution of Bayer type color area scan image sensor 400.In the case of images captured by diagonal CFA image sensor 1700 (FIG.17), the resolution may be tripled in both dimensions as compared to thecolor pixel resolution of diagonal CFA image sensor 1700.

FIG. 19 illustrates two exemplary color area scan image sensors 1900 and1950 having multiple regions with different color filter arrayproperties. Color area scan image sensors 1900 and 1950 areadvantageously implemented in an event timing system such as system 100(FIG. 1).

Color area scan image sensor 1900 includes three regions: region 1910(1)configured with a color filter array optimized for high-resolution TDI,and regions 1910(2) and 1910(3) configured with a Bayer-type colorfilter array as discussed in connection with FIG. 4. In certainembodiments, region 1910(1) is located on the optical axis of theimaging objective used to form images on color area scan image sensor1900. Thus, a TDI image generated from images captured by region 1910(1)may form an ideal side view of an object of interest travelling in adirection orthogonal to the optical axis. For example, color area scanimage sensor 1900 is implemented as area scan image sensor 110 in system100 (FIG. 1), and region 1910(1) is located on the optical axis ofimaging objective 120. TDI image information may be extracted fromregion 1910(1) while regions 1910(2) and 1910(3) provide standardtwo-dimensional images and/or additional TDI images.

Color area scan image sensor 1950 includes two regions: region 1960(1)configured with a color filter array optimized for high-resolution TDI,and region 1960(2) configured with a Bayer-type color filter array asdiscussed in connection with FIG. 4. In certain embodiments, region1960(1) is located on the optical axis of the imaging objective used toform images on color area scan image sensor 1950. Thus, a TDI imagegenerated from images captured by region 1960(1) may form an ideal sideview of an object of interest travelling orthogonal to the optical axis.For example, color area scan image sensor 1950 is implemented as areascan image sensor 110 in system 100 (FIG. 1), and region 1960(1) islocated on the optical axis of imaging objective 120. This requiresshifting the center of color area scan image sensor 1950 away from theoptical axis of imaging objective 120. TDI image information may beextracted from region 1960(1) while region 1960(2) provides standardtwo-dimensional images and/or additional TDI images.

Color area scan image sensor 1900 and 1950 may be modified to includemore regions and/or regions of other color filter array configurations,in addition to the respective TDI dedicated regions 1910 and 1920,without departing from the scope hereof.

FIG. 20 shows one exemplary system 2000 for image capture and,optionally, timing of events using a sensor 2010. In an embodiment,system 2000 incorporates system 100 of FIG. 1. Sensor 2010 is incommunication with a data processing system 2020 through interface 150(FIGS. 1 and 2). Optionally, TDI module 140 (FIG. 1) performs TDIprocessing of images captured by sensor 2010, through imaging optics2012, and communicates TDI images to data processing system 2020 throughinterface 150. Sensor 2010, or optional TDI module 140, may time stampimages using time from clock 160. In one embodiment, sensor 2010 is anarea scan image sensor, for example a CMOS area scan image sensor. Inanother embodiment, sensor 2010 is a line scan sensor. In yet anotherembodiment, sensor 2010 is color area scan image sensor 400 of FIG. 4 orcolor area scan image sensor 1400 of FIG. 14. In a further embodiment,sensor 2010 is trilinear color image sensor 1500 of FIG. 15. In anadditional embodiment, sensor 2010 is filtered area scan image sensor1000 of FIG. 10. Sensor 2010, imaging optics 2012, optional TDI module140, interface 150, and optional clock 160 may be integrated in a camera2015. Data processing system 2020 includes a processor 2030, memory2040, and an input/output interface 2050. Memory 2040 includes a datastorage 2041, for storing images sent to data processing 2020 frominterface 150 and results of processing performed by data processing2020. Memory 2040 further includes algorithms 2042, implemented asmachine-readable instructions in a memory 2040, for processing of imagesreceived from interface 150. In an embodiment, algorithms 2042 arelocated in a non-volatile portion of memory 2040. In another embodiment,data processing system 2020 retrieves algorithms 2042 from anon-volatile memory, located externally to data processing system 2020,and stores algorithms 2042 to a volatile portion of memory 2040.Input/output interface 2050 provides two-way communication with a user.

In certain embodiments, input/output interface 2050 is a wirelessinterface. For example, input/output interface 2050 is a WiFi orBluetooth interface. In this embodiment, a mobile device, such as acellular phone or a smartphone, may be used to control camera 2015and/or receive data therefrom. This mobile device may function as dataprocessing system 2020, or be a separate control device 2016.

Optionally, system 2000 includes an alternate event timing system 2060.Alternate event timing system 2060 includes an event recorder 2062 and,optionally, an alternate clock 2064. Alternate event timing system 2060detects and identifies events and assigns a time to each such eventusing a clock. In an embodiment, time is provided by alternate clock2064. In another embodiment, time is provided by clock 160. Alternateevent timing system 2060 may not be based on imaging of the events butuse other forms of event detection. In one embodiment, alternate eventtiming system 2060 provides timing at greater or lesser accuracy thanthat provided by the camera based system composed of camera 2015, clock160, and optional TDI module 140. Alternate clock 2064 may be based on aGlobal Positioning System (GPS) time signal. A GPS based embodiment ofclock 2064 has particular utility when system 2000 is operated inconjunction with other event timing systems, such that these may besynchronized with each other.

In certain embodiments, alternate event timing system 2060 is based on aradio-frequency identification. Objects, e.g., race participants, aretagged with a radio-frequency identification (RFID) chip. Event recorder2062 and alternate clock 2064 are a radio-frequency timing system thatdetects and identifies RFID chips when they come into proximity to eventrecorder 2062.

FIG. 21 illustrates one exemplary embodiment of system 2000 of FIG. 20,in which alternate event timing system 2060 is an RFID-based eventtiming system 2160 that includes an RFID decoder 2165. Objects aretagged with RFID chips 2170 that are detected and identified by RFIDdecoder when in proximity. RFID-based event timing system 2160 receivestime from clock 160 associated with camera 2010, eliminating the needfor synchronization of two separate clocks.

FIG. 22 is a flowchart illustrating one exemplary method 2200 forprocessing a series of input images, captured at an input frame rate,and associated times to generate a series of output images,corresponding to an arbitrary frame rate, and associated times. Theimages are provided by an event recording and timing system, e.g.,systems 100, 600, 2000 or 2100 of FIGS. 1, 6, 20 and 21, respectively.Method 2200 may be used to modify the time resolution of an image basedevent timing system subsequent to image capture. In an embodiment,method 2200 is implemented in data processing system 2020 (FIGS. 20 and21) as frame rate adjust algorithm 2043 and executed by processor 2030of data processing system 2020.

A series of input images, captured at an input frame rate, and timingare received from, e.g, interface 150 of system 2000 (FIG. 20) or 2100(FIG. 21) in a step 2210. In a step 2220, an output frame rate isselected. In one example of step 2220, a user specifies an output framerate. This output frame rate is communicated to data processing system2020 of system 2000 (FIG. 20) or 2100 (FIG. 21) through input/outputinterface 2050. In a step 2225, an initial output time series isdetermined, where the initial output time series corresponds to imagescaptured at the output frame rate selected in step 2220.

Steps 2230 through 2260 are repeated for all initial output times. Astep 2230 evaluates the initial output time under consideration. If theinitial output time is identical to an input time, method 2200 proceedsto step 2240, wherein the output image is set to equal the input imageassociated with the input time. If the initial output time is notidentical to an input time, method 2200 proceeds to a step 2250. In step2250, the output image associated with the initial output time iscalculated as a weighted average of input images captured close to theinitial output time. In an embodiment, the output image is calculated asa weighted average of two input images: the input image captured nearestthe initial output time and prior thereto and the input image capturednearest the initial output time and subsequent thereto. The weights ofthe weighted average may decrease with increasing time differencebetween the initial output time and the input time associated with inputimages contributing to the weighted average. From both step 2240 and2250, method 2200 proceeds to a step 2260. In step 2260, a final outputtime is assigned to the output image generated in either step 2240 orstep 2250. The final output time is set to equal the latest of the inputtimes associated with input images contributing to the output image.Steps 2225 through 2260 may be executed by processor 2030 of FIGS. 20and 21 according to instructions in frame rate adjust algorithm 2043(FIGS. 20 and 21). In a step 2270, the output images and associatedfinal output times are outputted, for example to a user or computersystem by input/output interface 2050 (FIGS. 20 and 21).

FIG. 23 is a flowchart illustrating one exemplary method 2300 forautomatically reducing the amount of image data generated by an eventtiming system utilizing image capture, such as systems 100 (FIG. 1), 600(FIG. 6), 2000 (FIG. 20), and 2100 (FIG. 21). Method 2300 may beimplemented in data processing system 2020 (FIGS. 20 and 21) asalgorithm crop image series 2044. In a step 2310, a series of image andassociated times are provided, for example by interface 150 (FIGS. 1,20, and 21). In a step 2320, the correspondence between events, such asa race participant crossing the finish line, and times are provided. Thecorrespondence provided in step 2320 may be generated by processor 2030(FIGS. 20 and 21) according to instructions in a correlator algorithm2045 (FIGS. 20 and 21). In one embodiment, events are identified byalternate event timing system 2060 (FIG. 20) or RFID based event timingsystem 2160 (FIG. 21). In another embodiment, events are identified byTDI module 140 (FIGS. 1, 20, and 21) using edge detection.

After performing steps 2310 and 2320, method 2300 proceeds to step 2330,wherein events of interest are selected. Events of interest may bepredefined as, e.g., the first N events (where N is a specified,positive integer), events associated with certain RFIDs, or eventsassociated with the occurrence of multiple events within a short timeframe. In a step 2340, the image series is cropped by removing imagesnot associated with an event of interest, e.g., images captured aspecified time interval before or after the time associated with theevent of interest. Steps 2330 and 2340 may be performed exclusively byprocessor 2030 (FIGS. 20 and 21) based on the instructions embedded incrop image series algorithm 2044, or in combination with user inputprovided through input/output interface 2050 (FIGS. 20 and 21). A step2350 outputs the cropped images series generated in step 2040. In anembodiment, step 2350 is performed by input/output interface 2050 (FIGS.20 and 21).

In embodiments where events are identified in real time, using one ormore of alternate event timing system 2060 (FIG. 20), RFID based eventtiming system 2160 (FIG. 17), and TDI module 140 (FIGS. 1, 20, and 21),method 2300 may be performed only at times when events are identified.For example, TDI module 140 (FIGS. 1, 20, and 21) may include a circularbuffer. TDI module 140 may evaluate the circular buffer using edgedetection. Upon detection of an edge, indicative of an event, therelevant input series is communicated to data processing system 2020(FIGS. 20 and 21) for execution of steps 2340 and 2350.

FIG. 24 illustrates one exemplary scenario 2400 and associated methodfor capturing images of a moving object 2420 using an image sensor withfour lines 2410(1), 2410(2), 2410(3), and 2410(4). In an embodiment,lines 2410(i) are pixel lines of an area scan sensor. In certainembodiments, lines 2410(i) are pixel lines selected from a larger numberof pixel lines of an area scan sensor. In the scenario illustrated inFIG. 24, four frames 2401, 2402, 2403, and 2404 are captured as afunction of time (2415) while object 2420 moves across the image fieldassociated with lines 2410(1), 2410(2), 2410(3), and 2410(4). The lines2410(i) are oriented perpendicular to the direction of motion (2425) ofobject 2420.

For illustration purposes, object 2420 is segmented into four areas A,B, C, D of equal size in the dimension parallel to the direction ofmotion of object 2420. The frame rate at which frames 2401, 2402, 2403,and 2404 are captured is matched to the speed of object 2420, such thatthe image of each of areas A, B, C, D shifts by one line 2410(i) betweeneach frame. Specifically, as object 2420 moves, area A is imaged ontoline 2410(1) in frame 2401, line 2410(2) in frame 2402, line 2410(3) inframe 2403, and line 2410(4) in frame 2404.

TDI may be performed by integrating lines across frames while takinginto account the frame-to-frame shifts of the captured image of object2420. An enhanced image of area A of object 2420 is formed byintegrating line 2410(1) of frame 2401, line 2410(2) of frame 2402, line2410(3) of frame 2403, and line 2410(4) of frame 2404. The exampleillustrated in FIG. 24 is non-limiting and is readily extended to anynumber of lines 2410(i), any number of frames, any number of objects,and any number of areas. In an embodiment, frames 2401, 2402, 2403, and2404 are captured by area scan image sensor 110 of FIG. 1. In certainembodiments, TDI is performed off-sensor, e.g., by TDI module 140 (FIG.1), using for example method 300 of FIG. 3. In a further embodiment,increased resolution is achieved using systems 400 (FIG. 4), 1400 (FIG.14) or 1500 (FIG. 15) and associated methods.

In one embodiment, frames 2401, 2402, 2403, and 2404 are captured by aninterline charge coupled device (CCD) area scan sensor with lines2410(1), 2410(2), 2410(3), and 2410(4). In an interline CCD area scansensor, the process of reading out pixel charges imposes no delaybetween integration of different frames. Each pixel of the interline CCDarea scan sensor has an associated masked pixel. The readout process isinitiated by a reset operation that shifts all pixel charges accumulatedduring integration of one frame to the corresponding masked pixels, andintegration of the next frame follows immediately after the resetoperation. The light collection efficiency of the interline CCD areascan sensor is therefore 100%, assuming that the delay associated withthe reset operation is negligible.

In another embodiment, lines 2410(1), 2410(2), 2410(3), and 2410(4)belong to a CMOS area scan image sensor. CMOS area scan image sensorsmay be configured with either a global shutter or a rolling shutter. Theintegration and readout process of a global shutter CMOS area scan imagesensor is analogous to that of an interline CCD. Rolling shutter CMOSarea scan image sensors may be implemented with a global reset or arolling reset, where rolling reset is the more commonly availableconfiguration. FIG. 25 illustrates the integration and readout process2500 for a rolling shutter CMOS area scan image sensor implemented withglobal reset. FIG. 26 illustrates the integration and readout process2600 for a rolling shutter CMOS area scan image sensor implemented withrolling reset. In a rolling shutter CMOS area scan image sensorimplemented with global reset, all pixels are reset at the same time andthen read out line by line. The pixels are not allowed to integrateduring readout, which means that the sensor is inactive during thereadout process. In a rolling shutter CMOS area scan image sensorimplemented with rolling reset, individual pixel rows are reset and readout on a rolling basis. While one row is being read out, all other rowsare still integrating. When readout of the one row is completed, it isagain allowed to integrate and readout of the next row is initiated.

In order to compare the light collection efficiency of the two resettypes, it is assumed that the integration time equals the readout timefor both types. In the embodiment illustrated in FIG. 26, the four lines2410(1), 2410(2), 2410(3), and 2410(4) coincide with pixel rows. Hence,pixels are read out line by line. Accordingly, in a case of N lines of arolling shutter CMOS area scan image sensor, implemented with rollingreset and running at its maximum frame rate, N readout periods arecompleted in a full frame cycle. With equal readout and integrationtimes, each line integrates for a duration equivalent to N readoutperiods before being read out. Applying the same assumptions to arolling shutter CMOS area scan image sensor implemented with globalreset, the embodiment illustrated in FIG. 25, yields that the sensorspends half a frame cycle integrating and half a frame cycle readingout.

In FIG. 25, the line status for lines 2410(1), 2410(2), 2410(3) and2410(4) is indicated as a function of time 2415. It is assumed that theduration of the reset operation is negligible and the line status istherefore either “integrate” (INT) or “readout” (READ). The image ofeach segment of object 2420 shifts by one line during a frame cycle, asindicated by segment A which is imaged onto line 2410(1) during frame2401 (label 2520(1)), line 2410(2) during frame 2402 (label 2520(2)),2410(3) during frame 2403 (label 2520(3)), and line 2410(4) during frame2404 (label 2520(4)). Frame 2401, for example, consists of readoutsignals 2530(1), 2530(2), 2530(3), and 2530(4), all resulting from asynchronized integration.

Time delay integration can be performed by integrating lines asdiscussed for FIG. 24. The result is equivalent to that obtained with aninterline CCD area scan image sensor, or global shutter CMOS area scanimage sensor, except that the light collection efficiency is 50%. Therolling shutter CMOS area scan image sensor with global reset may forexample be implemented in systems 100 (FIG. 1), 200 (FIG. 2), 600 (FIG.6), 2000 (FIG. 20), or 2100 (FIG. 21). Time delay integration may forexample be performed using method 300 of FIG. 3.

In FIG. 26, the line status for the lines 2410(1), 2410(2), 2410(3) and2410(4) is indicated as a function of time 2415. As for FIG. 25, it isassumed that the duration of the reset operation is negligible and theline status is therefore either “integrate” (INT) or “readout” (READ).The image of each segment of object 2420 shifts by one line during aframe cycle. However, in this case, not all lines are read out at thesame time. An exemplary frame consists of readout signals 2630(1),2630(2), 2630(3), and 2630(4). These readout signals result fromasynchronous integration. While the readout signal for line 2410(1) isaligned with segment positions, the readout signals for lines 2410(2),2410(3), and 2410(4) are increasingly shifted therefrom. Likewise, assegment A shifts from line to line, indicated by labels 2620(1),2620(2), 2620(3), and 2620(4), the corresponding readout signal containsan increasing contribution from segment B. However, the frame rate maybe adjusted to compensate for the asynchronous integration such that arolling shutter image sensor implemented with rolling reset may be usedwithout degradation of the TDI images generated therefrom. For example,the frame rate at which images are captured may be increased, ascompared to the nominal frame rate of a global shutter image sensor,such that the image of a passing object moves by one line in theduration of one frame time plus one readout time.

A benefit of the rolling shutter CMOS area scan image sensor implementedwith rolling reset is that the light collection efficiency mayapproximate 100%. For a rolling shutter CMOS area scan image sensor withN lines and rolling reset, the readout time associated with a line isonly 1/(N+1) of the frame cycle duration. The light integration dutycycle is therefore N/(N+1). In the embodiment with four lines,illustrated in FIG. 26, the light integration duty cycle is 80%.However, for a sensor with, e.g., 1024 lines, the light integration dutycycle is 99.9%.

Time delay integration can be performed by integrating lines asdiscussed for FIG. 24. The result is equivalent to that obtained with aninterline CCD area scan image sensor, or a global shutter CMOS area scanimage sensor, except for a small sub-frame blur and a slight decrease inlight integration duty cycle. The rolling shutter CMOS area scan imagesensor with rolling reset may be implemented in, e.g., systems 100 (FIG.1), 600 (FIG. 6), 2000 (FIG. 20), or 2100 (FIG. 21). Time delayintegration may for example be performed using, e.g., method 300 of FIG.3.

FIG. 27 illustrates one exemplary camera system 2700 configured with analignment assistance system. Camera system 2700 includes a camera 2710coupled with a mount 2720 that has at least four-axis movement includingthree orthogonal rotational degrees of freedom and one translationaldegree of freedom. Camera 2710 includes system 100 of FIG. 1, a level2712, and an optional alignment control system 2714. Camera 2710 may beimplemented without TDI module 140 of system 100 (FIG. 1) withoutdeparting from the scope hereof. Camera 2710 is associated with acoordinate system defined by three orthogonal axes 2730, 2740, and 2750.The coordinate system is fixed relative to camera 2710 such that itmoves with camera 2710. Axis 2750 is parallel to a vertical direction inimages captured by camera 2710. The origin of the coordinate system,i.e., the intersect of axes 2730, 2740, and 2750, may be located withincamera 2710 or externally thereto. Mount 2720 is configured to provideat least rotation 2731 about axis 2730, rotation 2741 about axis 2740,rotation 2751 about axis 2750, and translation 2742 along axis 2740.

In an exemplary use scenario, camera system 2700 is used to captureimages of a finish line. The direction of the finish line and thedirection of the gravitational force together define a finish plane.Alternatively, the finish plane is defined by the direction of thefinish line and another direction that is generally perpendicular to thedirection of motion of race participants crossing the finish line. Mount2720 is used to align camera 2710 to be level, as indicated by level2712, such that the direction of the gravitational force is vertical inimages 115 (FIG. 1) captured by camera 2712. This corresponds to axis2750 being parallel to the direction of the gravitational force. Notethat in this example camera 2710 has been placed relative to the finishline such that translation along axis 2740 results in generallyleft-right movement of the finish line in images captured by camera2710. Mount 2720 is further used to place camera 2710 in the finishplane, such that the finish line is vertical in images 115 (FIGS. 1 and2). FIGS. 28 through 31, discussed below, illustrate two methods forperforming this alignment.

In one embodiment, level 2712 is an electronic level and mount 2720includes motorized actuation. Level 2712 is communicatively coupled toan alignment control system 2714. Alignment control system 2714 isfurther communicatively coupled to system 100 and mount 2720. Alignmentcontrol system 2714 processes measurements by level 2712 and imagescaptured by system 100. Alignment control system 2714 controls mount2720 accordingly to achieve the desired alignment of camera 2710. Thisembodiment facilitates automatic alignment of camera 2710.

In another embodiment, alignment camera 2710 is aligned manually by anoperator using measurements by level 2712 and images captured by system100. In yet another embodiment, alignment control system 2714 controls aportion of the degrees of freedom of mount 2720, while other degrees offreedom are controlled by an operator. In this embodiment, the operatormay be aided by instructions provided by control system 2714. Forexample, alignment control system 2714 controls mount 2720 to controlrotations 2731, 2741, and 2751, and, as needed, provides instructions toan operator for adjusting translation 2742.

Camera 2710 may include camera 2015 of FIG. 20, instead of system 100,without departing from the scope hereof. Additionally, camera 2710 maybe a camera that does not have TDI functionality without departing fromthe scope hereof.

FIG. 28 illustrates one exemplary method 2800 for aligning camera 2710of camera system 2700 (FIG. 27) with a finish line. FIG. 28 is bestviewed together with FIG. 27 and FIG. 29. Method 2800 may be performedmanually or automatically, or a combination thereof, as discussed above.In a step 2810, mount 2720 rotates camera 2710 about axes 2730 and 2740to level camera 2710. This corresponds to making axis 2750 parallel tothe direction of the gravitational force. For example, alignment controlsystem 2714 receives measurements from level 2712 and controls mount2720 to level camera 2710. In a step 2820, camera 2710 captures an imageof a scene that includes the finish line. For example, alignment controlsystem 2714 triggers camera 2710 to capture an image 115 (FIGS. 1 and2). An exemplary image 2910(1) including a finish line image 2920 (1) isillustrated in FIG. 29. In a step 2830, the finish line is detected inthe image captured in step 2820. For example, alignment control system2714 detects the finish line image 2920(1) in image 2910(1). In anotherexample, an operator identifies the finish line image 2920(1) in image2910(1) and provides the identified location to alignment control system2714. In a step 2840, the finish line image 2920(1) in image 2910(1) isused to calculated the rotation 2751 about axis 2750 and translation2742 along axis 2740 required to place camera 2710 in the finish plane.For example, alignment control system 2714 analyzes the location andorientation of finish line 2920(1) in image 2910(1) to determinerotation 2751 and translation 2742 required to place camera 2710 in thefinish plane. This may include utilizing knowledge of the distance fromcamera 2710 to a specified point in the imaged scene. In a step 2850,mount 2720 rotates and translates camera 2710 according to the output ofstep 2840. Mount 2720 performs rotation 2751 and translation 2742. Forexample, alignment control system 2714 controls mount 2720 to performrotation 2751 and translation 2742. A resulting image 2910(2), if one iscaptured, is illustrated in FIG. 29. The finish line image 2920(2) isvertical in image 2910(2).

FIG. 30 illustrates another exemplary method 2600 for aligning camera2710 of camera system 2700 (FIG. 27) with a finish line. FIG. 30 is bestviewed together with FIG. 27 and FIG. 31. Method 3000 may be performedmanually or automatically, as discussed above. Method 3000 begins withperforming step 2810 of FIG. 28. In a subsequent step 3020, mount 2720rotates camera 2710 about axis 2750 while the position of the finishline in images 115 (FIGS. 1 and 2) is monitored. FIG. 31 illustrates anexemplary image 3110(1) captured prior to performing this rotation. Inimage 3110(1), the finish line image 3120(1) is located in theright-hand portion of image 3110(1). Mount 2720 rotates camera 2710until the finish line is horizontally centered in image 115 (FIGS. 1 and2). This is illustrated in FIG. 31 as exemplary image 3110(2) whereinthe finish line image 3120(2) is horizontally centered. For example,alignment control system 2714 continuously analyzes images 115 (FIGS. 1and 2) captured by system 100 while controlling mount 2720 to rotatecamera 2710 as needed. In a step 3030, mount 2720 translates camera 2710along axis 2750, while the position of the finish line in images 115(FIGS. 1 and 2) is monitored, until the finish line is vertical. When avertical finish line image is achieved, camera 2710 is located in thefinish plane. FIG. 31 illustrates an exemplary image 3110(3), with avertical finish line image 3120(3), captured after performing thistranslation. For example, alignment control system 2714 continuouslyanalyzes images 115 (FIGS. 1 and 2) captured by system 100 whilecontrolling mount 2720 to translate TDI camera 2010 as needed.

FIG. 32 illustrates one exemplary system 3200 for generating anddisplaying scoreboard-type video using an event timing system with animage sensor and a display. Scoreboard-type video includes, for example,results lists, standings, images generated by a TDI camera or otherphoto-finish system, video, commercials, and other graphics. System 3200is an embodiment of system 2000 of FIG. 20. System 3200 includes acamera 3215, which is an embodiment of optional camera 2015 (FIG. 20),data processing system 2020 (FIG. 20), and a display 3260. System 3200may farther include alternate event timing system 2060 (FIG. 20). Camera3215 includes an image sensor 3210, imaging optics 2012 (FIG. 20), aninterface 3250, and a video generator 3220 for generatingscoreboard-type video. In certain embodiments, image sensor 3210 is anarea scan image sensor, such as area scan image sensor 110 (FIG. 1).Optionally, camera 3215 further includes TDI module 140 (FIG. 1) and/orclock 160 (FIG. 1). Video generator 3220 includes a memory 3240. Memory3240 includes a results data storage 3242 for storing results datagenerated by data processing system 2020 and received by video generator3220 through interface 3250. Additionally, memory 3240 includesmachine-readable instructions 3244 encoded in memory 3240. In anembodiment, machine-readable instructions 3244 are located in anon-volatile portion of memory 3240. In another embodiment, videogenerator 3220 retrieves machine-readable instructions 3244 from anon-volatile memory, located externally to video generator 3220, andstores machine-readable instructions 3244 to a volatile portion ofmemory 3240. Video generator 3220 further includes a processor 3230 forprocessing of results data 3242, according to instructions 3244, toproduce scoreboard-type video. Video generator 3220 communicates thescoreboard-type video to display 3260 through interface 3250.Instructions may be communicated to video generator from a user or anexternal computer system, e.g., data processing system 2020, viainterface 3250 and stored to instructions 3244. Such instructionsinclude, for example, typographical settings, graphical settings, andoverall screen layout. Interface 3250 may include communication portsfor communicating the scoreboard-type video to other displays such as acomputer, or a network of computers. Interface 3250 may include one ormore wireless communication ports.

System 3200 provides a simple and cost-effective alternative toconventional scoreboard generation, which is based on separate timingsystem(s) and generation of scoreboard data. Conventionally, scoreboarddata is generated using a scoreboard with an integrated scoreboardcontroller or an external scoreboard controller. The scoreboardcontroller receives results from a timing system, processes the resultsusing scoreboard controller software, and generates video for thescoreboard. In contrast, system 3200 utilizes video generationcapability integrated in the timing system, specifically in camera 3215,for generating scoreboard type video. The scoreboard type video iscommunicated directly to display 3260 through interface 3250. Interface3250 may include a High-Definition Multimedia Interface (HDMI) and/or awireless communication port, for this purpose. The wirelesscommunication port may be a Wi-Fi communication port, for examplecapable of communicating scoreboard type video to a wireless-to-HDMIconverter communicatively coupled with an HDMI port of display 3260.Thus, system 3200 eliminates the need for a scoreboard and scoreboardcontroller. Since many commercially available area scan image sensorsinclude video generation capability, the electronic elements of camera3215 may be based on affordable and readily available electroniccomponents. In an embodiment, display 3260 is a Light Emitting Diode(LED) display.

The video generation capability of system 3200 may be employed duringalignment of camera 3215 with respect to a scene. In an embodiment,camera 3215 is configured to communicate images captured by image sensor3210 directly to video generator 3220. Video generator 3220 may processa stream of such images to generate scoreboard-type video including thestream of images. This scoreboard-type video may be communicated todisplay 3260 via interface 3250, such that an operator may align camera3215 by watching a real-time image stream on display 3260.

FIG. 33 illustrates one exemplary method 3300 for generating anddisplaying scoreboard-type video using an event timing system withintegrated video generation capability. Method 3300 may be performedusing system 3200 of FIG. 32. In a step 3310, event timing data isgenerated using a camera system. In one embodiment, the camera system isa TDI camera system, such as system 100 (FIG. 1) or camera system 3215(FIG. 32), and event timing data includes images, such as TDI images 145(FIGS. 1 and 2). In another embodiment, the event timing data includesarea scan images, such as digital two-dimensional images 115 (FIGS. 1and 2) or two-dimensional images captured by area scan image sensor 110(FIGS. 1 and 32). Step 3310 is performed, for example, by camera 3215(FIG. 32).

In a step 3320, event timing data is communicated to a data processingsystem. For example, camera 3215 (FIG. 32) communicates event timingdata, such as images, to data processing system 2020 (FIGS. 20 and 32)via interface 3250 (FIG. 32).

In a step 3330, the data processing system processes the event timingdata, communicated thereto in step 3320, to generate results data. In anembodiment, the results data include event timing results determined byanalyzing images, such as TDI images 145 (FIGS. 1 and 2) received fromthe event timing system. For example, processor 2030 (FIGS. 20 and 32)of data processing system 2020 (FIGS. 20 and 32) processes imagesreceived from camera 3215 (FIG. 32) in step 3320 according to algorithms2042 (FIGS. 20 and 32) to generate event timing results. The eventtiming results may be stored to data storage 2041.

In a step 3340, the results data generated in step 3330 are communicatedto the camera system. For example, data processing system 2020 (FIGS. 20and 32) communicates the results data to interface 3250 of camera 3215(FIG. 32).

In a step 3350, the results data are processed by the camera to generatescoreboard-type video. The camera processes the results data usingonboard video generation capability. For example, video generator 3220(FIG. 32) processes results data received from interface 3250 (FIG. 32)to generate scoreboard-type video. Processor 3230 (FIG. 32) stores datareceived from interface 3250 (FIG. 32) to results data 3242 (FIG. 32).Processor 3230 (FIG. 32) then retrieves and processes the results datafrom results data 3242 (FIG. 32) according to instructions 3244. Thescoreboard type video may include other elements based on other datathan the results data generated in step 3330 without departing from thescope hereof. For example, the scoreboard type video may include imagescaptured by the image sensor, such as a live stream of images.

In a step 3360, the scoreboard-type video generated in step 3350 iscommunicated to a display. For example, camera 3215 (FIG. 32)communicates scoreboard-type video generated by video generator 3220(FIG. 32) to display 3260 (FIG. 32) via interface 3250 (FIG. 32). Thescoreboard-type video may be streamed to display 3260 (FIG. 32) as it isgenerated or temporarily stored to memory 3240 (FIG. 32) andcommunicated to display 3260 (FIG. 32) at a later time. Memory 3240(FIG. 32) may function as a buffer that ensures continuous streaming.

FIG. 34 illustrates one exemplary event timing system 3400 that uses asingle data processing system 2020 (FIG. 20) to generate results basedon data received from multiple separate cameras 2015 (FIG. 20) and/orother data generating systems. In an embodiment, system 3400 includesone or more alternate event timing systems 2060 (FIG. 20). In certainembodiments, system 3400 includes at least one camera 3215 (FIG. 32)capable of generating scoreboard-type video, and at least one display3260 (FIG. 32) for displaying the scoreboard-type video. Optionally,system 3400 includes one or more alternate measuring systems 3410 forproviding event results that are not time-based. For example, alternatemeasuring system 3410 is a system for measuring distance, such as thelength of a jump or a throw. System 3400 may include any number ofcameras 2015, alternate event timing systems 2060, alternate measuringsystems 3410, and cameras 3215 communicatively coupled to dataprocessing system 2020, without departing from the scope hereof. System3400 may further include any number of displays 3260 communicativelycoupled to one or more cameras 3215 without departing from the scopehereof.

In an exemplary use scenario, cameras 2015 and, optionally alternateevent timing systems 2060, cameras 3215, and alternate measuring systems3410 are employed in a sports event that includes several individualevents occurring concurrently or sequentially. For example, a track andfield event typically includes a variety of running competitions,jumping competitions, throwing competitions. Each one of thesecompetitions has associated needs for measuring results using one ormore of cameras 2015 and, optionally alternate event timing systems2060, cameras 3215, and alternate measuring systems 3410. Frequently,multiple displays 3260 are installed in the stadium area to displaydifferent types of results.

Processes disclosed herein as being performed by a TDI module includedin a camera, for example TDI module 140 (FIG. 1), in certainembodiments, may alternatively be either fully or partly performed byanother processing system external to the camera, for example dataprocessing system 2020 (FIG. 20), without departing from a scope hereof.Such a data processing system may receive captured images and processthese at any later point in time. The camera may be equipped with a datacompression module for reducing the data rate associated with export ofcaptured (as opposed to TDI) images. Likewise, processes disclosedherein as being performed by a data processing system external to acamera, such as data processing system 2020 (FIG. 20), may alternativelybe either fully or partly performed by a TDI module included in thecamera, such as embodiments of TDI module 140 (FIG. 1), or by a anotherdata processing module included in the camera.

Features described above as well as those claimed below may be combinedin various ways without departing from the scope hereof. For example, itwill be appreciated that aspects of one system or method for processingevent timing images described herein may incorporate or swap features ofanother system or method for processing event timing images describedherein. The following examples illustrate possible, non-limitingcombinations of embodiments described above. It should be clear thatmany other changes and modifications may be made to the methods anddevice herein without departing from the spirit and scope of thisinvention:

(A1) A system for processing event timing images may include (a) an areascan image sensor for generating sequential digital two-dimensionalimages of a scene, and (b) a time delay integration module forprocessing the sequential digital two-dimensional images to generate atime delay integration image of a moving object in the scene.

(A2) In the system denoted as (A1), the time delay integration modulemay be separate from the area scan image sensor.

(A3) In the systems denoted as (A1) and (A2), the area scan image sensormay be implemented in a camera, and the time delay integration modulemay be separate from the camera.

(A4) In the systems denoted as (A1) through (A3), the area scan imagesensor may be a CMOS image sensor.

(A5) In the systems denoted as (A1) through (A4), the area scan imagesensor may be a CMOS image sensor with a rolling shutter.

(A6) In the system denoted as (A5), the CMOS image sensor with a rollingshutter may be implemented with rolling reset.

(A7) In the systems denoted as (A1) through (A6), the scene may includea moving object, the sequential two-dimensional images may includelines, and the area scan image sensor may have a frame ratecorresponding to object image movement at a rate of one line persequential digital two-dimensional image.

(A8) In the systems denoted as (A1) through (A6) of claim 1, the scenemay include a moving object, the sequential two-dimensional images mayinclude lines, and the area scan image sensor may have a frame ratecorresponding to object image movement at a rate of half a line persequential digital two-dimensional image.

(A9) In the systems denoted as (A1) through (A8), the scene may includea finish line of a race and the moving object may include a raceparticipant, or a portion of a race participant.

(A10) The systems denoted as (A1) through (A9) may further include (a) acamera with the area scan image sensor and a level, and (b) anadjustable mount coupled with the camera.

(A11) The system denoted as (A10) may further include an alignmentcontrol system for automatically adjusting the mount to align the camerawith respect to a finish line.

(A12) In the systems denoted as (A10) and (A11), the mount may includethree mutually orthogonal, rotational degrees of freedom and onetranslation degree of freedom.

(A13) In the systems denoted as (A1) through (A12), the time delayintegration module may include image processing circuitry, implementedin a field programmable gate array, wherein the image processingcircuitry may be adapted for processing the sequential digitaltwo-dimensional images to generate the time delay integration image

(A14) In the systems denoted as (A1) through (A13), the area scan imagesensor may include color pixels, where each color pixel is composed of aplurality of photosites

(A15) In the system denoted as (A14), the time delay integration imagemay be a color time delay integration image.

(A16) In the system denoted as (A15), the image processing circuitry maybe adapted for processing individual ones of the photosite signals togenerate the color time delay integration image with greater resolutionthan resolution of the sequential digital two-dimensional images.

(A17) In the systems denoted as (A1) through (A16), the time delayintegration module may include image processing circuitry adapted forsegmenting at least a portion of each of the sequential digitaltwo-dimensional images into input lines and forming the time delayintegration image from integrals of input lines, wherein each input lineof an integral corresponds to a different one of the sequential digitaltwo-dimensional images.

(A18) In the system denoted as (A17), the number of input lines of atleast one integral may be non-integer.

(A19) The systems denoted as (A17) and (A18) may further include acontroller communicatively coupled to the time delay integration module,and the image processing circuitry may be adapted to adjust the numberof input lines, according to signals received from the controller, toadjust the brightness of the time delay integration image.

(A20) In the system denoted as (A19), the image processing circuitry maybe further adapted to independently adjust, for individual pixels of thetime delay integration image, the number of input lines.

(A21) In the system denoted as (A20), the number of input lines may benon-integer for at least a portion of the time delay integration image.

(A22) In the systems denoted as (A21), the area scan image sensor mayinclude a filter having a plurality of filter portions with a respectiveplurality of transmissions, and the sequential digital two-dimensionalimages may include a plurality of image portions having a respectiveplurality of brightnesses, wherein each image portion corresponds to oneof the filter portions.

(A23) In the systems denoted as (A17) through (A22), the area scan imagesensor being a color sensor, wherein each color sensor pixel includes aBayer type array of photosites, and the input lines may alternatebetween (a) original pixels composed of signals from photosites from thesame line image frame captured by the color sensor and (b) crossoverpixels composed of signals from photosites from two sequentiallycaptured line image frames, to produce input lines at twice theresolution of the sequential digital two-dimensional imagesperpendicular to the input lines.

(A24) In the systems denoted as (A17) through (A22), the area scan imagesensor may be a color sensor with a plurality of trilinear color lines,wherein each of the trilinear color lines includes first, second, andthird photosite lines having a respective first, second, and third colorsensitivity, and the input lines may alternate between (a) originalpixels composed of signals from first, second, and third photosite linesbelonging to a first line image frame captured by the area scan imagesensor, (b) first crossover pixels composed of signals from the firstline image frame and a subsequently captured second line image frame,wherein the crossover pixels comprise signals from two photosite linesof the first line image frame and one photosite line of the second lineimage frame, and (c) second crossover pixels composed of signals fromthe first line image frame and the second line image frame, the secondcrossover pixels comprising signals from one photosite line of the firstline image frame and two photosite lines of the second image line image,to produce input lines at three times the resolution of the sequentialdigital two-dimensional images perpendicular to the input lines.

(A25) In the systems denoted as (A1) through (A24), the area scan imagesensor may be a color sensor, and the time delay integration image mayinclude original color pixels of the color sensor and crossover colorpixels formed by combining photosites from different images captured bythe color sensor.

(A26) In the systems denoted as (A1) through (A25), the area scan imagesensor and the time delay integration module may be integrated in acamera that is communicatively coupled with (a) a data processing systemfor generating results data from images received from the camera, and(b) a display for displaying scoreboard-type video, wherein the camerafurther includes a video generator for processing the results data togenerate the scoreboard-type video.

(B1) A method for processing event timing images may include (a)capturing sequential digital two-dimensional images of a scene using anarea scan image sensor, and (b) processing the sequential digitaltwo-dimensional images to generate a time delay integration image of anobject moving in the scene.

(B2) The method denoted as (B1), may further include communicating thesequential digital two-dimensional images from the area scan imagesensor to a module, separate from the area scan image sensor, forperforming the step of processing.

(B3) In the methods denoted as (B1) and (B2), the area scan image sensormay be implemented in a camera, and the module may be separate from thecamera.

(B4) In the methods denoted as (B1) through (B3), the step of processingmay include integrating the sequential digital two-dimensional images toform a time delay integration image of at least a portion of a movingobject in the scene by (a) segmenting at least of portion of each of thesequential digital two-dimensional images into input lines, and (b)populating each line of the time delay integration image with anintegral over a plurality of input lines, each of the plurality of inputlines being selected from a different one of the sequential digitaltwo-dimensional images to substantially match the movement of the movingobject in a direction perpendicular to the input lines.

(B5) In the method denoted as (B4), the step of processing may furtherinclude adjusting the number of input lines to adjust brightness of thetime delay integration image.

(B6) In the method denoted as (B5), the step of adjusting the number ofinput lines may include independently adjusting, for each pixel of thetime delay integration image, the number of input lines to locallyadjust the brightness of the time delay integration image.

(B7) In the methods denoted as (B4) through (B6), the number of inputlines may be non-integer for at least a portion of the time-delayintegration image.

(B8) In the methods denoted as (B1) through (B7), the area scan imagesensor may be a color sensor.

(B9) In the method denoted as (B8), the step of processing thesequential digital two-dimensional images may further include increasingresolution of the time delay integration image by including crossoverpixels formed by combining photosites from sequentially capturedtwo-dimensional images.

(B10) In the methods denoted as (B1) through (B9), the scene may includea moving object, the two-dimensional images may include lines, and thestep of capturing may include capturing images at a frame ratecorresponding to object image movement at a rate of one line persequential image.

(B11) In the methods denoted as (B1) through (B9), the scene may includea moving object, the two-dimensional images may include lines, and thestep of capturing may include capturing images at a frame ratecorresponding to object image movement at a rate of half a line persequential image.

(B12) In the methods denoted as (B1) through (B11), the scene mayinclude a finish line of a race and the object may include a participantin the race, or a portion of a participant in the race.

(B13) In the methods denoted as (B1) through (B12), the area scan imagesensor may include a rolling shutter.

(B14) In the methods denoted as (B1) through (B12), the area scan imagesensor may include a rolling shutter implemented with rolling reset.

(B15) In the methods denoted as (B1) through (B14), the area scan imagesensor may include a filter having a plurality of filter portions with arespective plurality of transmissions, and the step of processing mayfurther include selecting an portion of the sequential digitaltwo-dimensional images associated with one of the filter portions togenerate a time delay integration image of a certain brightness.

(C1) A method for processing a plurality of input images associated witha respective plurality of input times, wherein the input images andinput times are provided by an event timing system, may include (a)selecting an output frame rate, (b) generating a plurality of outputimages, corresponding to the output frame rate, from the plurality ofinput images, and (c) assigning to each output image a final output timeprovided by the event timing system, wherein the final output time isthe input time associated with an input image contributing to the outputimage.

(C2) The method denoted as (C1) may further include determining aninitial output time series corresponding to the output frame rate.

(C3) In the method denoted as (C2), in the step of generating, eachoutput image may be identical to an input image, when the initial outputtime is identical to an input time, and a weighted average of inputimages captured close to the initial output time, when the initialoutput time is not identical to an input time.

(C4) In the method denoted as (C3), the weighted average may be aweighted average of the two input images with associated input timesnearest the initial output time, wherein one of the two input images iscaptured before the initial output time and the other of the two inputimages is captured after the initial output time, when the initialoutput time is not identical to an input time.

(C5) In the method denoted as (C4), weights of the weighted average maydecrease with increasing time difference between the initial output timeand the input time associated with the input images contributing to theweighted average.

(C6) The methods denoted as (C1) through (C5) may further includegenerating the input images using any one of the systems denoted as (A1)through (A26).

(C7) In the method denoted as (C6), the input images may be time-delayintegration images generated by any one of the systems denoted as (A1)through (A26).

(C8) In the method denoted as (C6), the input images may be digitaltwo-dimensional images captured by the area scan image sensor of any oneof the systems denoted as (A1) through (A26).

(C9) The methods denoted as (C1) through (C7) may further includegenerating the input images as time delay integration images accordingto any one of the methods denoted as (B1) through (B15).

(D1) A method for processing images and associated event times providedby an event recording and timing system may include: (a) receiving (i)images and associated times and (ii) a correspondence between times andevents, (b) selecting events of interest, and (c) automaticallydiscarding images not associated with an event of interest, using aprocessor and machine-readable instructions.

(D2) In the method denoted as (D1), the correspondence between times andevents may be provided by a radio frequency identification timingsystem.

(D3) In the methods denoted as (D1) and (D2), the images may be timedelay integration images.

(D4) The methods denoted as (D1) through (D3) may further includegenerating the images using any one of the systems denoted as (A1)through (A26).

(D5) In the method denoted as (D4), the images may be time-delayintegration images generated by any one of the systems denoted as (A1)through (A26).

(D6) The method denoted as (D3) may further include generating the timedelay integration images according to any one of the methods denoted as(B1) through (B15).

(E1) A system for recording and timing of events may include (a) acamera system for capturing images of the events and comprising a clock,(b) an event recorder for detecting the events and being communicativelycoupled with the clock, and (c) a data processing system capable ofassigning times provided by the clock to the images captured by thecamera system and events detected by the event recorder.

(E2) In the system denoted as (E1), the data processing system mayinclude a processor and machine-readable instructions encoded innon-volatile memory, wherein the instructions are adapted for, whenexecuted by the processor, assigning the times.

(E3) In the systems denoted as (E1) and (E2), the camera system mayinclude a CMOS image sensor.

(E4) In the system denoted as (E3), the CMOS image sensor may include arolling shutter.

(E5) In the system denoted as (E4), the rolling shutter may beimplemented with rolling reset function.

(E6) The systems denoted as (E1) through (E5) may further include a timedelay integration module having circuitry for performing time delayintegration of the images captured by the camera system.

(E7) In the system denoted as (E6), the circuitry being implemented in afield programmable gate array.

(E8) In the systems denoted as (E6) and (E7), the time delay integrationmodule may be the time delay integration module of any one of thesystems denoted as (A1) through (A26).

(E9) In the systems denoted as (E1) through (E2), the camera may utilizethe area scan image sensor of any one of the systems denoted as (A1)through (A26) to capture the images of the events.

(E10) In the systems denoted as (E1) through (E9), the event recordermay be a radio-frequency identification decoder for detecting andidentifying radio-frequency identification chips in proximity.

(E11) In the systems denoted as (E1) through (E10), the data processingsystem may include instructions adapted for, when executed by theprocessor, correlating images of events captured by the camera systemwith events detected by the event recorder.

(E12) In the systems denoted as (E1) through (E11), the data processingsystem may include instructions adapted for, when executed by theprocessor, discarding images captured by the camera system notassociated with an event detected by the event recorder.

(F1) An area scan image sensor may include a plurality of color pixels,wherein each color pixel includes three different photosite typessensitive to three different colors, the photosites being arranged in a3×3 array such that each row and each column of 3×3 array comprises thethree photosite types and every row and column has photositeconfiguration different from any other row and column, respectively.

(F2) In the area scan image sensor denoted as (F1), the three photositetypes may have sensitivity to red, green, and blue light respectively.

(F3) Each of the area scan image sensors denoted as (F1) and (F2) may beimplemented in an event timing system for providing time delayintegration images from images captured by the area scan image sensor.

(F4) Each of the area scan image sensor denoted as (F1) and (F2) may beimplemented in an event timing system for providing time delayintegration images, from images captured by the area scan image sensor,at a resolution that is increased compared to the resolution of colorpixels of the area scan image sensor.

(F5) In the area scan image sensors denoted as (F3) and (F4), the eventtiming system may be any one of the systems denoted as (A1) through(A22), (A25), (A26), and (E1) through (E12).

(G1) A system for processing event timing images may include (a) acamera comprising (i) an area scan image sensor for capturing images ofa scene including a line and (ii) a level, (b) an adjustable mountcoupled with the camera, and (c) an alignment control system forautomatically adjusting the mount to align the camera with respect tothe line.

(G2) In the system denoted as (G1), the line may be a finish line of arace.

(G3) In the systems denoted as (G1) and (G2), the camera may furtherinclude a time delay integration module for processing images capturedby the area scan image sensor to generate a time delay integrationimage.

(G4) In the system denoted as (G3), the area scan image sensor may bethe area scan image sensor of any one of the systems denoted as (A1)through (A26).

(G5) In the systems denoted as (G3) and (G4), the time delay integrationmodule may be the time delay integration module of any one of thesystems denoted as (A1) and (A26).

(H1) A system for processing event timing images may include (a) acamera comprising an image sensor for capturing images and a videogenerator for generating scoreboard type video, and (b) a dataprocessing module, communicatively coupled with the camera, forgenerating results data from images received from the camera andcommunicating the results data to the video generator.

(H2) The system denoted as (H1), may further include a display fordisplaying the scoreboard-type video, wherein at least a portion of thescoreboard-type video is generated from the results data.

(H3) In the system denoted as (H1) and (H2), the camera may include atime delay integration module, communicatively coupled with the imagesensor, for processing images captured by the image sensor to generatetime delay integration images.

(H4) In the system denoted as (H3), the time delay integration modulemay be the time delay integration module of any one of the systemsdenoted as (A1) through (A26).

(H5) In the systems denoted as (H1) through (H4), the image sensor maybe an area scan image sensor and the images may be two-dimensional areascan images.

(I1) A software product includes instructions, stored on non-transitorycomputer-readable media, wherein the instructions, when executed by acomputer, perform steps for processing sequential digitaltwo-dimensional images of a scene comprising a moving object to form atime delay integration image, wherein the software product may include:(a) instructions for segmenting at least of portion of each of thesequential digital two-dimensional images into input lines, and (b)instructions for populating each line of the time delay integrationimage with an integral over a plurality of input lines, wherein each ofthe plurality of input lines being selected from a different one of thesequential digital two-dimensional images to substantially match themovement of the moving object in a direction perpendicular to the inputlines.

(I2) In the software product denoted as (I1), the instructions forprocessing sequentially captured digital two-dimensional images mayfurther include instructions for adjusting the number of input lines toadjust brightness of the time delay integration image.

(I3) In the software product denoted as (I2), the instructions foradjusting the number of input lines may include instructions forindependently adjusting, for each pixel of the time delay integrationimage, the number of input lines to locally adjust the brightness of thetime delay integration image.

(I4) In the software product denoted as (I3), the number of input linesmay be non-integer for at least a portion of the time-delay integrationimage.

(I5) In the software products denoted as (I1) through (I4), thesequential digital two-dimensional images may be color images.

(I6) In the software product denoted as (I5), the instructions forprocessing the sequential digital two-dimensional images may furtherinclude instructions for increasing resolution of the time delayintegration image by including crossover pixels formed by combiningphotosite signals from sequentially captured two-dimensional images.

(I7) The software products denoted as (I1) through (I6) may beimplemented in any one of the systems denoted as (A1) through (A26).

(J1) A software product includes instructions, stored on non-transitorycomputer-readable media, wherein the instructions, when executed by acomputer, perform steps for processing a plurality of input imagesassociated with a respective plurality of input times, the input imagesand input times being provided by an event timing system, wherein theinstructions may include: (a) instructions for selecting an output framerate, (b) instructions for generating a plurality of output images,corresponding to the output frame rate, from the plurality of inputimages, and (c) instructions for assigning to each output image a finaloutput time provided by the event timing system, wherein the finaloutput time being the input time associated with an input imagecontributing to the output image.

(J2) In the software product denoted as (J1), the instructions forprocessing a plurality of input images may further include instructionsfor determining an initial output time series corresponding to theoutput frame rate, and the instructions for generating a plurality ofoutput images may include instructions for setting each output image toequal an input image, when the initial output time is identical to aninput time, and a weighted average of input images captured close to theinitial output time, when the initial output time is not identical to aninput time.

Changes may be made in the above methods and systems without departingfrom the scope hereof. It should thus be noted that the matter containedin the above description and shown in the accompanying drawings shouldbe interpreted as illustrative and not in a limiting sense. Thefollowing claims are intended to cover generic and specific featuresdescribed herein, as well as all statements of the scope of the presentmethod and system, which, as a matter of language, might be said to falltherebetween.

What is claimed is:
 1. A method for modifying frame rate of an imageseries generated by an event timing system, comprising: receiving, fromthe event timing system and via an input/output interfacecommunicatively coupled to a processor, a plurality of input images anda respective plurality of different input times generated by the eventtiming system, the input times corresponding to an input frame rate forrecording of the input images by the event timing system, all of theinput images being captured by the same camera at the respectivedifferent input times; receiving, via the input/output interface, anoutput frame rate that is different from the input frame rate;determining, by using the processor to execute machine-readableinstructions stored in non-volatile memory, an initial output timeseries corresponding to the output frame rate; generating, from theplurality of input images and by using the processor to executemachine-readable instructions stored in non-volatile memory, a pluralityof output images having the output frame rate, each of the output imagesbeing (a) identical to an input image when the initial output time isidentical to an input time or (b) a weighted average of input imagescaptured close to the initial output time when the initial output timeis not identical to an input time, the weighted average being a weightedaverage of the two input images with associated input times nearest theinitial output time, one of the two input images being captured beforethe initial output time and the other of the two input images beingcaptured after the initial output time, when the initial output time isnot identical to an input time; assigning, by using the processor toexecute machine-readable instructions stored in non-volatile memory, toeach output image a final output time received from the event timingsystem, the final output time being one of the input times associatedwith an input image contributing to the output image; and outputting theoutput images together with each respective final output time via theinput/output interface.
 2. The method of claim 1, wherein weights of theweighted average decrease with increasing time difference between theinitial output time and the input time associated with the input imagescontributing to the weighted average.
 3. A non-transitorycomputer-readable media comprising instructions that, when executed by acomputer, perform steps for modifying frame rate of an image seriesgenerated by an event timing system, the instructions comprising:instructions for receiving, from the event timing system and via aninput/output interface communicatively coupled to the processor, aplurality of input images and a respective plurality of different inputtimes generated by the event timing system, the input timescorresponding to an input frame rate for recording of the input imagesby the event timing system, all of the input images being captured bythe same camera at the respective different input times; instructionsfor receiving, via the input/output interface, an output frame rate thatis different from the input frame rate; instructions for determining aninitial output time series corresponding to the output frame rate;instructions for processing the plurality of input images to generate aplurality of output images having the output frame rate, each of theoutput images being (a) identical to an input image when the initialoutput time is identical to an input time or (b) a weighted average ofinput images captured close to the initial output time when the initialoutput time is not identical to an input time, the weighted averagebeing a weighted average of the two input images with associated inputtimes nearest the initial output time, one of the two input images beingcaptured before the initial output time and the other of the two inputimages being captured after the initial output time, when the initialoutput time is not identical to an input time; instructions forassigning to each output image a final output time received from theevent timing system, the final output time being one of the input timesassociated with an input image contributing to the output image; andinstructions for outputting the output images together with eachrespective final output time via the input/output interface.
 4. Thenon-transitory computer-readable media of claim 3, the input imagesbeing line images.
 5. The method of claim 1, the input images being lineimages.